Systems and methods for controlling operation of an on-board component

ABSTRACT

Systems and methods allow a user to interact with an augmented reality device. In one implementation, a wearable device for controlling operation of an on-board component includes a display; at least one sensor configured to provide an output indicative of a viewing direction of a user; and at least one processing device. The at least one processing device is configured to cause at least one graphical icon associated with a control of at least one aspect of the on-board component to be shown on the display; determine, based on the output of the at least one sensor, whether the user is looking in a direction of the at least one graphical icon; and initiate the control of the at least one aspect of the on-board component when the user is determined to be looking in the direction of the at least one graphical icon.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 61/910,419, filed on Dec. 1, 2013, U.S.Provisional Patent Application No. 61/910,425, filed on Dec. 1, 2013,and U.S. Provisional Patent Application No. 62/043,759, filed on Aug.29, 2014, all of which are incorporated herein by reference in theirentirety.

TECHNICAL FIELD

The present disclosure relates generally to an augmented reality deviceand, more particularly, to methods and systems for representing andinteracting with augmented reality content using the augmented realitydevice.

BACKGROUND

Technology advances have enabled mobile personal computing devices tobecome more capable and ubiquitous. In many cases, these devices willhave both a display as well as a combination of sensors. For example,the devices may include GPS, accelerometers, gyroscopes, cameras, lightmeters, and compasses or some combination thereof. These devices mayinclude mobile computing devices as well as head mounted displays.

Additionally, these mobile personal computing devices are increasinglycapable of both displaying information for the user as well as supplyingcontextual information to other systems and applications on the device.Such contextual information can be used to determine the location,orientation and movement of the user interface display of the device.

SUMMARY

Embodiments consistent with the present disclosure provide an apparatusand methods for representing and interacting with augmented realitycontent.

Consistent with a disclosed embodiment, a wearable device provides avirtual menu to a user. The wearable device may include a display; atleast one sensor configured to provide an output indicative of anorientation of a head of the user; and at least one processing device.The at least one processing device may be configured to monitor anorientation of the head of the user based on the output of the at leastone sensor; determine based on the monitored orientation of the headwhether the user is looking upward or downward with respect to apredetermined horizontal threshold; and cause the virtual menu to beshown on the display if the user is determined to be looking upward ordownward with respect to the predetermined horizontal threshold.

Consistent with another disclosed embodiment, a method provides avirtual menu to a user of a wearable device. The method may includemonitoring, based on output of at least one sensor, an orientation of ahead of the user; determining based on the monitored orientation of thehead whether the user is looking upward or downward with respect to apredetermined horizontal threshold; and causing the virtual menu to beshown on a display of the wearable device if the user is determined tobe looking upward or downward with respect to the predeterminedhorizontal threshold.

Consistent with another disclosed embodiment, a wearable device providesa virtual menu to a user. The wearable device may include a display; atleast one sensor configured to provide an output indicative of a viewingdirection of the user; and at least one processing device. The at leastone processing device may be configured to cause a virtual menu to beshown on the display, the virtual menu including at least one selectableelement; monitor a viewing direction of the user based on the output ofthe at least one sensor, determine, based on the monitored viewingdirection, whether the user is looking in a direction of the at leastone selectable element of the virtual menu; determine an amount of timethat the user looks in the direction of the at least one selectableelement of the virtual menu; and cause at least one action associatedwith the at least one selectable element of the virtual menu if theamount of time exceeds a predetermined dwell time threshold.

Consistent with another disclosed embodiment, a method provides avirtual menu to a user of a wearable device. The method may includecausing a virtual menu to be shown on a display of the wearable device,wherein the virtual menu includes at least one selectable element;monitoring, based on output of at least one sensor, a viewing directionof the user, determining, based on the monitored viewing direction,whether the user is looking in a direction of the at least oneselectable element of the virtual menu; determining an amount of timethat the user looks in the direction of the at least one selectableelement of the virtual menu; and causing at least one action associatedwith the at least one selectable element of the virtual menu if theamount of time exceeds a predetermined dwell time threshold.

Consistent with another disclosed embodiment, a wearable device providesa virtual menu to a user. The wearable device may include a display; atleast one sensor configured to provide an output indicative of a viewingdirection of the user; and at least one processing device. The at leastone processing device may be configured to monitor a viewing directionof the user based on the output of the at least one sensor; provide amenu on the display; determine, based on the monitored viewingdirection, whether the user is looking in a direction of the location ofthe menu on the display; and expand the menu if the user is determinedto be looking in the direction of the location of the menu on thedisplay.

Consistent with another disclosed embodiment, a method provides avirtual menu to a user of a wearable device. The method may includemonitoring, based on output of at least one sensor, a viewing directionof the user; providing a menu on a display of the wearable device;determining, based on the monitored viewing direction, whether the useris looking in a direction of the location of the menu on the display;and expanding the menu if the user is determined to be looking in thedirection of the location of the menu on the display.

Consistent with a disclosed embodiment, a wearable device establishes acommunication path with a user of the wearable device and at least oneindividual. The wearable device includes a display; at least one sensorconfigured to provide an output indicative of a viewing direction of theuser; and at least one processing device. The at least one processingdevice is configured to monitor the viewing direction of the user basedon the output of the at least one sensor; determine, based on themonitored viewing direction, whether the user is looking in thedirection of the at least one individual; and establish thecommunication path between the user and the at least one individual ifthe user is determined to be looking in the direction of the at leastone individual.

Consistent with another disclosed embodiment, a method establishes acommunication path with a user of a wearable device and at least oneindividual. The method includes monitoring, based on an output of atleast one sensor, a viewing direction of the user; determining, based onthe monitored viewing direction, whether the user is looking in thedirection of the at least one individual; and establishing thecommunication path between the user and the at least one individual ifthe user is determined to be looking in the direction of the at leastone individual.

Consistent with a disclosed embodiment, a lockable, wearable device isprovided. The wearable device comprise a display; at least one sensorconfigured to provide an output indicative of a viewing direction of auser; and at least one processing device. The at least one processingdevice is configured to track a pattern of the viewing direction of theuser, and unlock the lockable, wearable device to provide the user withaccess to information on the display of the device when the trackedpattern of movement matches a predetermined pattern of movement.

Consistent with another disclosed embodiment, a lockable, wearabledevice is provided. The wearable device comprise a display, at least onesensor configured to provide an output indicative of a viewing directionof a user, and at least one processing device. The at least oneprocessing device is configured to cause an array of graphical objectsto be shown on the display of the wearable device; detect selection bythe user of at least two graphical objects from among the array ofgraphical objects based on the output indicative of the viewingdirection of the user; and unlock the lockable, wearable device toprovide the user with access to information on the display of the devicebased on whether the detected selection of the at least two graphicalobjects matches a predetermined object selection sequence.

Consistent with another disclosed embodiment, a method unlocks awearable device. The method includes tracking, using at least one sensorof the wearable device, a viewing direction of a user of the wearabledevice; and unlocking the wearable device to provide the user withaccess to information on a display of the wearable device when thetracked viewing direction matches a predetermined pattern of movement.

Consistent with another disclosed embodiment, a method unlocks awearable device. The method includes causing an array of graphicalobjects to be shown on a display of the wearable device; detectingselection by the user of at least two graphical objects from among thearray of graphical objects based on an output of at least one sensor ofthe wearable device, wherein the output is indicative of a viewingdirection of a user of the wearable device; and unlocking the wearabledevice to provide the user with access to information of a display ofthe device based on whether the detected selection of the at least twographical objects matches a predetermined object selection sequence.

Consistent with another disclosed embodiment, a wearable device providestask-based instructions to a user. The wearable device may include adisplay; a network interface; a data input device configured to captureinformation relating to progress of a task; and at least one processingdevice. The at least one processing device may be configured to provideto a server system, via the network interface, the captured informationrelating to progress of the task; receive from the server system, viathe network interface, information relating to a next step in the task;and cause the information relating to the next step in the task to beshown on the display.

Consistent with another disclosed embodiment, a method providestask-based instructions to a user of a wearable device. The method mayinclude capturing, via a data input device of the wearable device,information relating to progress of a task; providing to a serversystem, via a network interface, the captured information relating toprogress of the task; receiving from the server system, via the networkinterface, information relating to a next step in the task; and causingthe information relating to the next step in the task to be shown on adisplay of the wearable device.

Consistent with another disclosed embodiment, a wearable device controlsoperation of an on-board component. The wearable device may include adisplay; at least one sensor configured to provide an output indicativeof a viewing direction of a user; and at least one processing device.The at least one processing device may be configured to cause at leastone graphical icon associated with a control of at least one aspect ofthe on-board component to be shown on the display such that the userperceives the location of the at least one graphical icon as fixedrelative to real world coordinates; determine, based on the output ofthe at least one sensor, whether the user is looking in a direction ofthe at least one graphical icon; and initiate the control of the atleast one aspect of the on-board component when the user is determinedto be looking in the direction of the at least one graphical icon.

Consistent with another disclosed embodiment, a method controlsoperation of an on-board component of a wearable device. The method mayinclude causing at least one graphical icon associated with a control ofat least one aspect of the on-board component to be shown on a displayof the wearable device such that the user perceives the location of theat least one graphical icon as fixed relative to real world coordinates;determining, based on an output of at least one sensor of the wearabledevice configured to provide an output indicative of a viewing directionof the user of the wearable device, whether the user is looking in adirection of the at least one graphical icon; and initiating the controlof the at least one aspect of the on-board component when the user isdetermined to be looking in the direction of the at least one graphicalicon.

Consistent with another disclosed embodiment, a system interacts withand shares information between a plurality of users of a correspondingplurality of wearable devices. The system may include a networkinterface; and at least one processing device. The at least oneprocessing device may be configured to receive, via the networkinterface, information from a first wearable device; select from theplurality of wearable devices one or more designated wearable devices toreceive the information; and send, via the network interface, theinformation to the one or more designated wearable devices.

Consistent with another disclosed embodiment, a method interacts withand shares information between a plurality of users of a correspondingplurality of wearable devices. The method may include receiving, via anetwork interface, information from a first wearable device; selectingfrom the plurality of wearable devices one or more designated wearabledevices to receive the information; and sending, via the networkinterface, the information to the one or more designated wearabledevices.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processor and perform any of the methodsdescribed herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 illustrates an exemplary system for implementing disclosedembodiments.

FIG. 2 illustrates an exemplary Augmented Reality (AR) device consistentwith disclosed embodiments.

FIG. 3 is a block diagram illustrating a configuration of asoftware-driven system consistent with disclosed embodiments.

FIG. 4 illustrates an example of a user wearing an AR device consistentwith disclosed embodiments.

FIG. 5A illustrates an example of a virtual menu being displayed by anAR device consistent with disclosed embodiments.

FIG. 5B illustrates an example of displaying a sub-menu by an AR deviceconsistent with disclosed embodiments.

FIG. 6 is a flowchart of an exemplary process for providing a virtualmenu to a user by an AR device consistent with disclosed embodiments.

FIG. 7A illustrates an example of a selectable element of virtual menubeing displayed by an AR device by consistent with disclosedembodiments.

FIG. 7B illustrates another example of a selectable element of virtualmenu being displayed by an AR device consistent with disclosedembodiments.

FIG. 7C illustrates an example of expanding an element of virtual menubeing displayed by an AR device consistent with disclosed embodiments.

FIG. 8 is a flowchart of an exemplary process for causing an actionassociated with an element of virtual menu to be performed by an ARdevice consistent with disclosed embodiments.

FIG. 9A illustrates an example of a nested menu that is displayed by anAR device consistent with disclosed embodiments.

FIG. 9B illustrates another example of a nested menu that is displayedby an AR device consistent with disclosed embodiments.

FIG. 10 illustrates an example of accessing a nested menu that isdisplayed by an AR device consistent with disclosed embodiments.

FIG. 11 is a flowchart of an exemplary process for accessing a nestedmenu by an AR device consistent with disclosed embodiments.

FIG. 12 illustrates an example of a user of a wearable AR deviceinitiating communication with another user consistent with disclosedembodiments.

FIG. 13 illustrates an example of a user of a wearable AR deviceinitiating communication with a group of users consistent with disclosedembodiments.

FIG. 14 is a flowchart of an exemplary process for initiatingcommunicating with a user of a wearable AR device consistent withdisclosed embodiments.

FIG. 15 illustrates an example of a user unlocking a wearable AR deviceconsistent with disclosed embodiments.

FIG. 16A illustrates an example of a display showing a wearable ARdevice in a locked state consistent with disclosed embodiments.

FIG. 16B illustrates an example of a display showing a wearable ARdevice in an unlocked state consistent with disclosed embodiments.

FIGS. 17A-17K illustrate another example of a display showing a wearableAR device changing from a locked to an unlocked state consistent withdisclosed embodiments.

FIG. 18 is a flowchart of an exemplary process for unlocking a wearableAR device consistent with disclosed embodiments.

FIG. 19 illustrates an example of providing a user of an AR device withtask-based instructions consistent with disclosed embodiments.

FIG. 20 illustrates another example of providing a user of an AR devicewith task-based instructions consistent with disclosed embodiments.

FIG. 21 is a flowchart of an exemplary process for providing task-basedinstructions via an AR device consistent with disclosed embodiments.

FIG. 22 illustrates an example of a user wearing an AR device to controloperation of an on-board component consistent with disclosedembodiments.

FIG. 23A illustrates an example of a graphical icon associated withcontrolling an on-board component of an AR device consistent withdisclosed embodiments.

FIG. 23B illustrates an example of accessing a graphical icon associatedcontrolling an on-board component of an AR device consistent withdisclosed embodiments.

FIG. 23C illustrates an example of a menu of operations associated withan on-board camera associated with an AR device consistent withdisclosed embodiments.

FIG. 24 is a flowchart of an exemplary process for controlling anon-board component of an AR device consistent with disclosedembodiments.

FIG. 25 illustrates an example environment for sharing informationbetween users of AR devices consistent with disclosed embodiments.

FIG. 26 is a flowchart of an exemplary process for sharing informationbetween users of AR devices consistent with disclosed embodiments.

FIG. 27 is a flowchart of an exemplary process for selecting designatedAR devices to receive information consistent with disclosed embodiments.

DETAILED DESCRIPTION

Mobile personal computing devices may include one or more portabledisplays used to overlay virtual objects with real world objects.Virtual content that relates to one or more real world objects (e.g.,places, things, people, etc.) and that may be provided on a display maybe referred to as Augmented Reality (AR) content. Such AR content may beprovided on a display together with the real world objects to which theAR content relates. Further, the views of the real world objects on adisplay may correspond to computer-generated representations of thoseobjects or, in some embodiments (e.g., where at least a portion of thedisplay passes or channels light), may correspond to actual,non-computer-generated views of the objects.

In some embodiments, a device may provide various menus from which auser may select. For example, in one embodiment, a wearable augmentedreality device may provide a menu that appears to hover over a user'shead and outside of the user's field of view when the user is looking atthe horizon. To access the menu, the user looks up toward the spot wherethe user perceives the menu to be located. For example, in oneembodiment, the wearable augmented reality device may provide a userinterface that enables a user to select a menu item by looking at themenu item. In another embodiment, the wearable augmented reality devicemay provide a nested menu system that enables the user to look toward amenu shown on a display, select the menu, and expand the menu uponselection. In another embodiment, a system may provide the capability tomonitor the progress of a task assigned to a particular user of awearable augmented reality device. As steps in the task are completed,information relating to the next steps is passed to the user.

Users of AR systems may wish to control other devices. For example, inanother embodiment, a wearable augmented reality device may provide aninterface for controlling the operation of a camera without relying uponvoice or button activated controls.

Users of AR systems may also wish to communication with other users.Accordingly, in another embodiment, a system may allow a user of awearable augmented reality device to establish a communicationconnection with another individual or group of individuals usingwearable augmented reality devices by looking in the direction of thoseindividuals or group. In another embodiment, a one-to-many system mayenable sharing of information between multiple wearable augmentedreality device users. For example, information available to a particularuser can be made available to multiple other users by showing thatinformation on the displays of the multiple users.

Users of such devices may also wish for their content to be secure frombeing accessed by unauthorized users. Accordingly, in anotherembodiment, a wearable augmented reality device may provide a techniquefor locking and unlocking the device such that information stored on thedevice or conveyed to and from the device remains protected andaccessible only by the intended/authorized user.

FIG. 1 illustrates an exemplary system 100 for implementing thedisclosed embodiments. In one aspect, system 100 may include a serversystem 110, a user system 120, and network 130. It should be noted thatalthough a single user system 120 is shown in FIG. 1, more than one usersystem 120 may exist in system 100. Furthermore, although a singleserver system 110 is shown in FIG. 1, more than one server system 110may exist in system 100.

Server system 110 may be a system configured to provide and/or manageservices associated with providing AR content to users. Consistent withthe disclosure, server system 110 may provide AR content to user system120 based on information received from user system 120. For example,server system 110 may provide AR content to user system 120 based on adetected position, detected position change, or detected/determinedcontext of user system 120. Such AR content may also be provided to usersystem 120 based upon requests received from user system 120 or based ona detected and/or recognized object within a field of view of an imagingdevice associated with user system 120. In general, any type of ARcontent, including any desired form of information, can be provided fromserver system 110 to user system 120 based on the requirements of aparticular application. Further, any suitable trigger may be used toinitiate transfer and/or updating of AR content provided to user system120 by server system 110.

Server system 110 may include one or more components that performprocesses consistent with the disclosed embodiments. For example, serversystem 110 may include one or more computers, e.g., processor device111, a CPU, and/or other hardware-based applications processing devices,configured to execute software instructions programmed to performaspects of the disclosed embodiments. Such software instructions may bestored, for example, in database 113 and when implemented may createand/or maintain a global coordinate system, provide AR objects to usersystems for display, transmit information associated with the AR objectsto user system 120, or any other relevant function. In one aspect,database 113 may be co-located with server system 110. Alternatively,database 113 may be located remotely from the server system 110.Database 113 may include computing components (e.g., database managementsystem, database server, etc.) configured to receive and processrequests for data stored in memory devices of database(s) 113 and toprovide data from database 113.

User system 120 may include a system associated with a user (e.g., aconsumer, field technician, equipment operator, or any other individualthat may benefit from received AR content) that is configured to performone or more operations consistent with the disclosed embodiments. In oneembodiment, a user may operate user system 120 to perform one or moresuch operations. User system 120 may include a communication interface121, a processor device 123, a memory 124, a sensor array 125, a display122, and/or any other components that may facilitate the display of ARcontent to the user. The processor device 123 may be configured toexecute software instructions to perform aspects of the disclosedembodiments. User system 120 may be configured in the form of an ARdevice, such as a head mounted display (HMD). Although in the presentdisclosure user system 120 is described in connection with a HMD, usersystem 120 may include tablet devices, mobile phone(s), laptopcomputers, a wearable device, such as a smart watch, and any othercomputing device(s) known to those skilled in the art.

Communication interface 121 may include one or more communicationcomponents, such as cellular, WiFi, Bluetooth transceivers, or any otherwireless transceivers or communication equipment. Communicationinterface 121 may package and send user commands or input across network130 to server system 110. Based on these commands and/or input, serversystem may return additional or updated AR content to be displayed tothe user by user system 120. This additional or updated AR content, orinstructions for providing the additional or updated AR content, may bereceived from server system 110 via communication interface 121.Processor device 123 may access and use information received viacommunication interface 121.

Display 122 may include any suitable display for providing AR content toa user. For example, in some embodiments, display 122 may include one ormore translucent, transparent, or semi-transparent components configuredto pass at least some light from a scene to the eyes of a user. Display122 may also include opaque lenses or components, e.g., where the imagesseen by the user are computer-generated, constitute light-guidedprojections of real world objects, or include images captured by one ormore cameras (or other image acquisition devices) associated with usersystem 120. In such embodiments, generated, captured, and/orlight-guided images of real world objects may be projected onto opaqueor semi-opaque components associated with display 122. Display 122 mayalso project information using holographic images.

Sensor array 125 may include any type of sensor configured to provide anoutput signal useful in providing AR content to a user. In someembodiments, sensor array 125 may include one or more GPS sensors,cameras, barometric sensors, proximity sensors, physiological monitoringsensors, chemical sensors, magnetometers, gyroscopes, accelerometers,bar code scanners, motion detectors, image sensors, depth sensors, eyetracking sensors, etc.

Processor devices 111 and 123 may include one or more suitablehardware-based processing devices, such as a microprocessor, controller,central processing unit, etc. that can execute software-basedinstructions. In some embodiments, processor devices 111 and/or 123 mayinclude a microprocessor from the OMAP family manufactured by TexasInstruments, the Snapdragon™ family manufactured by Qualcomm™, or any ofvarious types of processor devices manufactured by other microprocessormanufacturers.

Consistent with disclosed embodiments, one or more components of system100, including server system 110 and user system 120, may also includeone or more memory devices (such as memories 112 and 124) as shown inexemplary form in FIG. 1. The memory devices may include magneticstorage units, optical storage units, RAM, ROM, hard drives, flashmemory, etc. Such memory units may be located together with serversystem 110 and/or user system 120 or may be located remotely withrespect to either or both systems. Memory 112 and or 124 may storesoftware instructions that are executed by processor devices 111 and123, such as one or more applications, network communication processes,operating system software, software instructions relating to thedisclosed embodiments, and any other type of application or softwareknown to be executable by processing devices. The memory devices may bea volatile or non-volatile, magnetic, semiconductor, tape, optical,removable, non-removable, or other type of storage device ornon-transitory computer-readable medium. The memory devices may be twoor more memory devices distributed over a local or wide area network, ormay be a single memory device.

In certain embodiments, the memory devices may include database systems,such as database storage devices, including one or more databaseprocessing devices configured to receive instructions to access,process, and send information stored in the storage devices. By way ofexample, database systems may include Oracle™ databases, Sybase™databases, or other relational databases or non-relational databases,such as Hadoop sequence files, HBase, or Cassandra.

In some embodiments, server system 110 and user system 120 may alsoinclude one or more additional components (not shown) that providecommunications with other components of system environment 100, such asthrough network 130, or any other suitable communicationsinfrastructure.

Network 130 may be any type of network that facilitates communicationsand data transfer between components of system environment 100, such as,for example, server system 110 and user system 120. Network 130 may be aLocal Area Network (LAN), a Wide Area Network (WAN), such as theInternet, and may include a single network or a combination of networks.Further, network 130 may include a single type of network or acombination of different types of networks, such as the Internet andpublic exchange networks for wireline and/or wireless communications.Network 130 may utilize cloud computing technologies that are familiarin the marketplace. Network 130 is not limited to the above examples,and system 100 may implement any type of network that allows theentities (and others not shown) included in FIG. 1 to exchange dataand/or information.

FIG. 2 illustrates an exemplary AR device 200, which may correspond toand include any of the components and features of user system 120. Atpresent, augmented reality glasses are the most common type of ARdevice. It is also possible to use a smart phone as an AR device.Therefore, it will be understood that the disclosed embodiments are notlimited to augmented reality glasses or any one type of AR device. Forexample, a relatively simple AR device might involve a projector with acamera interacting with the surrounding environment, where theprojection could be on a glass surface or on top of other objects.

As shown in FIG. 2, AR device 200 may include features relating tonavigation, orientation, location, sensory input, sensory output,communication and computing. For example, AR device 200 may include aninertial measurement unit (IMU) 201. IMU 201 may include axialaccelerometers or gyroscopes for measuring position, velocity andorientation. IMUs may enable determination of the position, velocityand/or orientation of AR device 200 within the surrounding real worldenvironment and/or its position, velocity and orientation relative toreal world objects within that environment in order to perform itsvarious functions.

AR device 200 may also include a Global Positioning System (GPS) unit202. GPS units receive signals transmitted by a plurality ofgeosynchronous earth orbiting satellites in order to triangulate thelocation of the GPS unit. In more sophisticated systems, the GPS unitmay repeatedly forward a location signal to an IMU to supplement theIMUs ability to compute position and velocity, thereby improving theaccuracy of the IMU. In the present case, AR device 200 may employ GPSto identify a location of AR device 200. In some embodiments, AR device200 may determine the user's presence and/or location using otherdevices in an environment of the user of AR device 200 (e.g., AR device200 may communicate with, for example, Bluetooth low energy beacons).

As mentioned above, AR device 200 may include a number of featuresrelating to sensory input and sensory output. AR device 200 may includeat least a front facing camera 203 to provide visual (e.g., video)input, a display (e.g., a translucent or a stereoscopic translucentdisplay) 204 to provide a medium for displaying computer-generatedinformation to the user, a microphone 205 to provide sound input andaudio buds/speakers 206 to provide sound output. In some embodiments,the visually conveyed digital data may be received by AR device 200through the front facing camera 203. In some embodiments, AR device 200may include one or more eye tracking sensors that may be configured tomonitor and/or track a viewing direction of a user based on the positionof one or both of the user's eyes, and provide an output relating to theviewing direction of the user (e.g., a direction of the user's gaze).

AR device 200 may also have communication capabilities, similar to othermobile devices, through the use of a cellular, WiFi, Bluetooth ortethered Ethernet connection. AR device 200 may also include an on-boardmicroprocessor 208. The on-board microprocessor 208 may control theaforementioned and other features associated with AR device 200.

In one exemplary embodiment, AR device 200 may include a see-throughdisplay and sensor systems that provide the device's location,orientation, and bearing (for example, latitude, longitude, altitude,pitch, roll or degree tilt from horizontal and vertical axes, andcompass heading). AR device 200 could be configured as glasses that canbe worn by a person. Further, one or more elements of the sensor systemmay be located on peripheral devices physically separate from thedisplay.

Additionally, in this embodiment, AR device 200 may rely on a computersoftware application to instruct the glasses to render virtual objectson the display field of view. Virtual objects include, but are notlimited to, text, images, models, icons. The user may view or interactwith virtual objects using the hardware and software applicationassociated with the AR glasses 200.

FIG. 3 is a system block diagram 300 illustrating a software-drivensystem associated with an AR device, in accordance with exemplarydisclosed embodiments. As illustrated, the software system may beconfigured into three layers. At the lowest layer is the augmentedreality device operating system 360. The operating system 360 may, forexample, be an Android based operating system, an iPhone based operatingsystem, a Windows Mobile operating system or the like. At the highestlayer may reside a third party applications layer 362. Applications thatare designed to work with the operating system 360 that either came withthe AR device or were loaded by the user may reside in this third layer.

The middle layer is referred to as the augmented reality shell 364. Theaugmented reality shell 364, as shown, includes a number of componentsincluding a command processor 366, an environmental processor 368, arendering services module 370, and a network interaction services module372. It should be understood that each of the functional modules and/orcomponents may be hardware, software, firmware or a combination thereof.

Environmental processor 368, in general, monitors the surrounding, realworld environment of the AR device (e.g., AR device 200) based on inputsignals received and processed by the AR device (e.g., from sensor array125). Environmental processor 368 may be implemented, as shown in FIG.3, similar to the other processing components, or it may be implementedseparately, for example, in the form of an application specificintegrated chip (ASIC). In one embodiment, the environmental processor68 may be configured to operate whenever the AR device is powered on.

Environmental processor 368, in turn, may also include severalprocessing modules: a visual processing module 374, a geolocationalprocessing module 376, and a positional processing module 378. Visualprocessing module 374 is primarily responsible for processing thereceived video, detecting and decoding the frames and processing thedigital data included with the video that was visually conveyed to theAR device.

Geolocational module 376 may receive and process signals relating to thelocation of the AR device. The signals may, for example, relate to GPScoordinates, the location of a WiFi hotspot, the proximity to one ormore local cell towers, or the proximity to one or more other devices(e.g., one or more Bluetooth low energy beacons).

Positional module 378 may receive and process signals relating to theposition, velocity, acceleration, direction and orientation of the ARdevice. Positional module 378 may receive these signals from an IMU(e.g., IMU 201). Positional processing module 378 may, alternatively oradditionally, receive signals from a GPS receiver, where it isunderstood that the GPS receiver can only approximate position (andtherefore velocity and acceleration) and where positional processingmodule 378 can then provide a level of detail or accuracy based on theGPS approximated position. Thus, for example, the GPS receiver may beable to provide the general GPS coordinates of a movie theater, butpositional processing module 378 may be able to provide the user'sorientation within the movie theater. Positional processing module 380may be employed in conjunction with visual processing module 374 tosynchronize user head movements with viewing experiences (e.g., whatrendering services module 370 will render on the display and, therefore,what the user sees).

Command processor 366 may process messaging between the modules and/orcomponents. For example, after visual processing module 374 processesthe digital data that was visually received through the video, visualprocessing module 374 communicates with the command processor 366 which,in turn, generates one or more commands to rendering services module 370to produce the computer generated data (e.g., text, graphics, additionalvideo, sound) that will be used to supplement the video and enhance theuser's viewing experience.

Rendering services module 370 may process the content of the digitaldata that was visually received and, based on instructions providedthrough command processor 366, generate and present (e.g., display) datain the form of sound, graphics/animation, text, additional video and thelike. The user can thus view the video and, in addition, experience thecomputer-generated information to supplement the video and enhance theviewing experience.

In some embodiments, the software applications associated with the ARdevice may create and maintain coordinate systems that correspond to theposition and orientation of the AR device, the field of view of the ARdevice, and the virtual content. These coordinate systems may correspondto locations in the real physical world or may be referenced withrespect to the AR device or to any other suitable reference. Themaintained coordinate systems may include a two dimensional Cartesiancoordinate system, a three dimensional Cartesian coordinate system, atwo dimensional Spherical coordinate system, a three dimensionalSpherical coordinate system, or any other suitable coordinate system.

AR content, including virtual objects or information, may be rendered atpositions on the display of the AR device defined by the softwareapplication. These positions may be relative to the user. For example,the initial position for display may be in the center of the user'sfield of view or at any other desired location with the user's field ofview. The AR content may also be rendered such that the user perceivesthe AR content as associated with a particular location or object in thereal world. For example, the virtual content may be rendered on thedisplay to appear to the user as located at coordinate positionscorresponding to a location in the real-world, for example, referring toa specific geographic coordinate, or to the current coordinate locationof another person or object. User movements relative to the specificgeographic coordinate, person, object, etc. may prompt the system toupdate the display of the AR content such that it appears snapped to thespecific geographic coordinate, person, object, etc.

If the user desires to move or “snap” the virtual content to a newposition, the user may select a virtual object, or group of objects,using an input method supported by the hardware and software applicationassociated with the AR device. For illustration purposes, one methodcould be to move a selection device, such as a mouse or pointer device.Another method would be to use gesture inputs (pointing of a finger,etc.), a push button device, touch sensitive surface, or voice commandsto augment selection. Another exemplary method would be to use a reticlerendered on the display that would allow head-movements made by the userto indicate the direction or location of the virtual object to beselected. Following the selection, the user may define a new positionfor the virtual object relative to the display of the AR device.

For example, a user of user system 120/AR device 200 may include a fieldtechnician, and the AR content to be displayed may include a servicemanual for a particular type of equipment. Initially, and by operationof a voice command, input device, etc., the technician may requestdisplay of AR content in the form of the service manual. In someembodiments, this information may be retrieved from server system 110 ormay be retrieved from a local memory 124 and may be displayed in display122. Initially, the AR content may be displayed in the center of theuser's field of view. Such positioning would enable natural access andviewing of the desired content. AR content in the center of the user'sfield of view, however, may be distracting or may impede the view of thereal world objects with which the technician needs to interact.Therefore, system 120 may be configured to re-position the AR content toanother location on display 122 or even to another location in virtualspace (maintained in memory) relative to the user (e.g., relative to theuser's head position).

Furthermore, the repositioned AR content can be accessed by the user.For example, the user may turn his head in the direction of therepositioned AR content and, based on the sensed motion of the head, therendering of the AR content may be updated relative to the motion of theuser's head. As another example, one or more eye tracking sensors may beconfigured to track a viewing direction of the user, such as the user'sgaze direction, and the rendering of the AR content may be updatedrelative to changes in the user's gaze direction. In some embodiments,the rendered AR content may appear to the user as fixed at therepositioned location, and the user may turn his head in the directionof the repositioned location in order to bring the AR content toward andinto the center of the field of view of display 122. Subsequent turningof the head away from the repositioned location will cause the renderedAR content on the display to move away from the center of the field ofview. In this way, the user (e.g., the technician) may move the desiredAR content to a virtual location that does not impede viewing of objectsin the center of the field of view, but that still allows access to thedesired AR content, for example, by turning toward the virtual positionof that content.

The user may define the new position of the virtual object by using aninput device or method supported by the hardware and software associatedwith AR device. For example, one method would be to “drag and drop” byfirst selecting the content to be moved, moving it by dragging it to thedesired new location and deselecting it to drop it in the new,repositioned location. Establishing a new position of the virtual objectmay create a new coordinate or change an existing coordinate associatedwith the virtual object in the software application of the AR device.

The user (or routines in the software) may define additional snappingproperties or rules associated with the virtual objects or newcoordinates. These properties may include parameters, tags, orinstructions for the software application of the AR device. For example,a property of the “snapped” virtual AR content may be to share the newcoordinates for the object with other users of system 100. Anotherexample may be to delete the new coordinates of the virtual objectsafter a period of time and then revert to the previous position. Yetanother example would be to lock the virtual object from being moved toanother coordinate by a different user of the AR device.

When the user initiates the steps to “snap” a virtual object to a newposition, the software application of the AR device may use informationfrom sensor systems to define a first coordinate system for the ARdevice and a second coordinate system for the virtual object that isselected by the user, which may be referred to as cardinal snappingherein.

As will be appreciated by a person skilled in the art having the benefitof this disclosure, numerous variations and/or modifications may be madeto the foregoing disclosed embodiments. For example, not all componentsare essential for the operation of AR device 200. Further, any componentmay be located in any appropriate part of AR device 200 and thecomponents may be rearranged into a variety of configurations whileproviding the functionality of the disclosed embodiments. Therefore, theforegoing configurations are examples and, regardless of theconfigurations discussed above, AR device 200 can provide variousfeatures that allow a user to interact and/or control AR device 200.Further details regarding various features provided by AR device 200 arediscussed below in further detail.

Virtual Overhead Menu

In one embodiment, a menu is provided for a wearable augmented realitydevice. The menu may appear to hover over a user's head and potentiallyjust outside of the user's field of view when the user is looking at thehorizon. To access the menu, the user may simply look up or down towarda spot where the user perceives the menu to be located.

This menu system may avoid the difficulties associated with menusprovided at the sides of a user's field of view. For example, whenpositioning a menu to the side of a user's field of view, it may bedifficult to determine whether a user turned his or her head to the sideto access a menu or whether the user merely wished to look in thatdirection without accessing the menu.

For example, consistent with an embodiment, a wearable device thatprovides a virtual menu to a user may include a display and at least onesensor configured to provide an output indicative of an orientation of ahead of the user. The wearable device may further include at least oneprocessing device configured to monitor an orientation of the head ofthe user based on the output of the at least one sensor. Based on themonitored orientation of the head, the at least one processing devicemay be further configured to determine whether the user is lookingupward or downward with respect to a predetermined horizontal threshold.If the user is determined to be looking upward or downward with respectto the predetermined horizontal threshold, the at least one processingdevice may further be configured to cause the virtual menu to be shownon the display.

FIG. 4 illustrates an example of a user wearing an AR device consistentwith disclosed embodiments. As shown in FIG. 4, user 401 is in anenvironment 400 and is wearing an AR device 200, as discussed aboveregarding FIG. 2. AR device 200 may include a frame configured to securethe device to the head of user 401. As discussed above, AR device 200may include sensor array 125. Sensor array 125 may include one or moresensors configured to detect a viewing direction of the user. In someembodiments, the one or more sensor may detect a viewing direction ofthe user by detecting the orientation of the head of user 401 bydetecting the orientation of the AR device 200. The one or more sensorsmay include any one or more of a motion detector (e.g., a gyroscopeand/or an accelerometer), a camera, a location determination device(e.g., a GPS device, an NFC reader), a magnetometer, and/or anorientation sensor (e.g., a theodolite, infra-red sensor). Such sensorsmay include magnetometers (compasses) to detect orientation or adirection that a user is looking, accelerometers (e.g., three axisaccelerometers) to detect the direction and speed of movementsassociated with AR device, or the like. For example, the orientation ofthe head of user 401 may be defined according to a coordinate system,such as a three-dimensional coordinate system (x, y, z) having theorigin at a point on AR device 200, such as a central point of display204, the position at which IMU 201 is located, or any other referencepoint on AR device 200. In other embodiments, one or more eye trackingsensors may be configured to track a viewing direction of the user bytracking and/or monitoring the eyes of the user to determine the user'sgaze direction.

In some embodiments, positional processing module 378 shown in FIG. 3may be configured to store instructions that when executed by aprocessing device (e.g., microprocessor 208) of AR device 200, determinethe orientation of the head of user 401 based on output from the one ormore sensors.

For example, in order to access a virtual menu, user 401 may move his orher head upward or downward, forming an angle α with respect to thehorizontal plane. The one or more sensors included in AR device 200 maybe configured to detect a viewing direction of the user. The sensors maythen output the detected viewing direction (e.g., the orientation of theuser and/or the user's gaze direction) to a processor device, such asmicroprocessor 208 provided in FIG. 2.

In some embodiments, the one or more sensors may be configured to detectthe orientation of the head of user 401 periodically. For example, theone or more sensors may be configured to detect the origination of thehead of user 401 at any suitable time interval (e.g., every second,multiple times per second (2, 4, 6, 10 or more times per second), oronce over the course of multiple seconds, etc.). Further, the one ormore sensors may be configured to output the detected angle of AR device200 with respect to the horizontal plane to a processing device (e.g.,microprocessor 208). Alternatively, the one or more sensors may beconfigured to output the detected coordinates of one or morepredetermined reference points of AR device 200 with reference to apredetermined coordinate system. For example, the one or more sensorsmay be configured to detect coordinates of multiple points of display204 in a predetermined three-dimensional coordinate system having anorigin set as the central point of AR device 200.

The processing device (e.g., microprocessor 208) of AR device 200 may beconfigured to monitor the orientation of the head of user 401 based onthe output of the one or more sensors. For example, the processingdevice may be configured to determine the orientation of the head ofuser 401 based on received coordinates of points of the device from theone or more sensors. In some embodiments, the processing device may beconfigured to monitor both the orientation of the head of user 401 andthe time duration that the head of user 401 stays in that orientationbased on output received from the sensor. For example, if the timeduration that the head of user 401 stays in a detected orientation isless than a predetermined time threshold, the processing device maydetermine that user 401 does not intend the detected orientation tocause AR device 200 to take an action. On the other hand, if the timeduration that the head of user 401 stays in a detected orientation isgreater than or equal to the predetermined time threshold, theprocessing device may determine that user 401 intends the detectedorientation to cause AR device 200 to take an action.

The processing device may be further configured to determine whetheruser 401 is looking upward or downward with respect to a predeterminedhorizontal threshold based on the detected orientation of the head ofuser 401. For example, the processing device may be further configuredto compare the detected orientation of the head of user 401 with thepredetermined horizontal threshold. If the detected orientation of thehead of user 401 is greater than or equal to the predeterminedhorizontal threshold, the processing device may be configured todetermine that the user is looking upward. On the other hand, if thedetected orientation of the head of user 401 is less than thepredetermined horizontal threshold, the processing device may beconfigured to determine that the user is not looking upward.

The predetermined horizontal threshold may be pre-configured by user 401through a user interface of AR device 200 or be pre-set based on adefault setting of AR device 200. For example, display 204 may display amenu with different values of the predetermined horizontal threshold toenable user 401 to make a selection. As another example, display 204 maydisplay a field that enables user 401 to input a desirable value of thepredetermined horizontal threshold. The predetermined horizontalthreshold may be set in units of degrees, radians, or any other units ofangular measurement. As an example, the predetermined horizontalthreshold may be set as 20, 30, or 60 degrees or more according to apreference of user 401.

As discussed above, in some embodiments, alternative or in addition tomonitoring an orientation of the user's head, the processing device ofAR device 200 may be configured to monitor a user's gaze direction basedon output received from one or more eye tracking sensors. For example,the processing device may be configured to determine whether user 401 islooking upward or downward with respect to a predetermined horizontalthreshold based on the direction of the user's gaze.

If the processing device determines that user 401 is looking upward ordownward with respect to the predetermined horizontal threshold, theprocessing device may cause a virtual menu to be shown on a display(e.g., display 204) of AR device 200. The virtual menu may be shown at apredetermined position on display 204, such as a central location ofdisplay 204 or at any other suitable location on display 204.

AR device 200 may be configured to display the menu either continuouslyor in response to a predetermined trigger. For example, in someembodiments, AR device 200 may display a virtual menu on display 204such that the menu is always visible within the field of view of theuser (e.g., at a location in an upper portion of the user's field ofview). In other embodiments, the menu may be displayed in response to atrigger, which may include, for example, a determination that the useris looking upward relative to a horizontal plane. In such embodiments,the menu may be caused to appear on the display in response to adetermination that the user is looking upward relative to a horizontalplane by an angle of 20 degrees, 30 degrees, 60 degrees, or more. Thus,regardless of the user's side to side head orientation, when the user isdetermined to be looking in an upward direction, the menu may be shownon the display. In some cases the user may be looking to the right andthen look upward to access the menu. In other cases, the user may belooking to the left and then look upward to access the menu. In stillother cases, the user may be looking forward and then look upward toaccess the menu.

Once the menu is provided on the display, it can remain in a fixedlocation relative to the display or may move relative to the display.For example, in some embodiments, the virtual menu may be locked in theuser's field of view, such as in a locked mode, while user 401 moves hisor her head to a new position. That is, once the menu has been providedon the display, the menu may be locked to a particular location relativeto the display such that subsequent movements of the head of the user donot cause movement of the menu relative to the display.

In other embodiments, the virtual menu may be provided at a fixed,perceived location relative to the user. For example, the user mayperceive that the virtual menu hovers above the user's head at aparticular angle relative to a horizontal looking direction. Thus, whilethe user looks left, right, or forward along a horizontal plane (orbelow a horizontal plane), the virtual menu may not appear on thedisplay. To access the menu, however, the user may simply look upwardabove the predetermined angular threshold (regardless of left-right headorientation), and the menu will be shown on the display. This mayprovide the user with the perception that the menu “hovers” at alocation above and forward of the user's forehead, for example. Once themenu is displayed, the menu location on the display may be unaffected bysubsequent left-right movement of the user's head. On the other hand,once the menu is shown on the display, subsequent changes in verticalorientation of the user's head (e.g., looking upward or downward, forexample) may cause changes in the location of the menu on the display.For example, once the menu is shown on the display, subsequent headmovements further in the upward direction may cause the location of themenu to move downward on the display. Similarly, subsequent headmovements in a downward direct may cause the location of the menu tomove upward on the display. Such behavior may be consistent with a userperception that the menu “hovers” above the user's head at a fixedvertical angle, regardless of horizontal angle. In this mode (e.g.,unlocked mode), once the head of user 401 is moved downward away fromthe perceived location of the menu, the virtual menu may disappear fromdisplay 204, and the virtual menu may reappear when the head of user 401is moved upward again. Of course, the processor device may be configuredto also cause changes in the left-right position of the virtual menurelative to the display in response to detected horizontal movements ofthe user's head (e.g., right-left changes in the looking direction ofthe user's head).

In some embodiments, AR device 200 may provide an interface for user 401to select the menu positioning mode of the virtual menu. For example, abutton may be placed on AR device 200 for user 401 to switch between thelocked and unlocked modes. As another example, selectable options forthe locked and unlocked modes may be displayed on display 204, and user401 may select a desirable mode to display the virtual menu.

FIG. 5A illustrates an example of a virtual menu being displayed by anAR device consistent with disclosed embodiments. For example, renderingservices module 370, discussed above in reference to FIG. 3, may storesoftware instructions that, when executed by a processing device (e.g.,microprocessor 208), cause audio messages to be played, and/or causetext messages and/or graphics to be displayed. As illustrated in FIG.5A, a virtual menu 502 is shown on display 204 of AR device 200 whenuser 401 is determined to be looking upward. As discussed above, display204 may include one or more lenses included on a pair of glasses, andvirtual menu 502 may be displayed on the lens in a semi-transparent formor opaque form. In some embodiments, the level of transparency of theshown menu may be selectable by the user.

Virtual menu 502 may include one or more selectable elements eachassociated with one or more actions that may be taken by AR device 200upon selection by user 401. For example, virtual menu 502 may presentoptions for user 401 to save AR content, share it with other users,rotate the display of the AR content, initiate a phone call, changesettings associated with a camera on AR device 200, and any otheroptions or tasks that the user may select to provide input, requestinformation, or initiate a task associated with AR device 200.

As noted, virtual menu 502 may be displayed on display 204 such that theuser perceives the virtual menu to be located at a predetermined, fixedangle of rise above the predetermined horizontal threshold.Additionally, initiation of the display of the virtual menu may occurbased on certain timing constraints. For example, in some embodiments,virtual menu 502 may be shown on display 204 after user 401 isdetermined to be looking upward with respect to a predeterminedhorizontal threshold for a time period that exceeds a predetermined timethreshold (e.g., 0.25 seconds, 0.5 seconds, 1 second, 2 seconds, or anyother suitable time). Accordingly, if user 401 looks upward with respectto the predetermined horizontal threshold for less than thepredetermined time threshold, the processing device may be configured tonot display virtual menu 502.

In some embodiments, the processing device of AR device 200 may befurther configured to determine whether user 401 is looking in adirection intersecting any of the selectable items of virtual menu 502and cause a selection of a particular item if the user looks in thedirection of the particular item for a time that exceeds a predeterminedtime threshold. For example, with reference to FIGS. 4 and 5A, if theprocessing device of AR device 200 determines that user 401 is lookingin a direction intersecting the item “save” on the virtual menu 502 fora time period that exceeds a predetermined threshold, the processingdevice may cause the item “save” on the virtual menu 502 to be selectedand take the corresponding action (e.g., saving data, such as an image).

FIG. 5B illustrates an example of displaying a sub-menu by an AR deviceconsistent with disclosed embodiments. In some embodiments, virtual menu502 may include a nested menu, and the processing device of AR device200 may be configured to cause a display of one or more sub-menuelements in response to a sensed selection of a higher level menuelement. As shown in FIG. 5B, virtual menu 502 is a nested menu, andsub-menu 503 may be displayed on display 204 in response to a sensedselection of a higher level menu item. For example, upon selection ofthe menu item “zoom in” of the virtual menu 502, the sub-menu 503 may bedisplayed containing different levels of zoom-in options. The virtualmenu may be a text menu, a graphical interface, or a combination of textand graphics.

The selection of the menu item or sub-menu item may be caused by user401 looking in the direction of the sub-menu item, touching theparticular menu item on the display, moving a selector to the positionof the particular menu item, intersecting a reticle with the menu item,generating a voice command, or the like. For example, user 401 may movea selector to the position of the particular menu item on the display tomake a selection by moving or rotating the position of the user's head.As another example, user 401 may touch the part of display 204 at whichthe menu item is shown to make a selection of the item. In anotherexample, user 401 may generate a voice command corresponding to the menuitem to make a selection. In some embodiments, the selection of the menuitem or sub-menu item may be caused by user 401 looking in a directionintersecting the particular item for a time period that exceeds apredetermined threshold. For example, a sensor of AR device 200 maysense that user 401 is looking in a direction intersecting a particularmenu item, and the processing device may in turn cause a selection ofthe item based on output received from the sensor. For example, theprocessing device of AR device 200 may cause the selection of a menuitem “zoom in” of the virtual menu 502 if user 401 looks in a directionintersecting the item “zoom in” for a time period that exceeds thepredetermined threshold. As another example, the processing device of ARdevice 200 may cause the selection of sub-menu item “2X” of the sub-menu503 if user 401 looks in a direction intersecting the item “2X” for atime period that exceeds the predetermined threshold. In someembodiments, the function of selecting an item by sight may be turned onor off by user 401. For example, a button may be placed on the AR devicefor user 401 to turn on or turn off the function of sight selection. Asanother example, a selectable option may be displayed on display 204 ofAR device 200 for user 401 to turn on or off the function of selectingan item by sight.

FIG. 6 is a flowchart of an exemplary process 600 for providing avirtual menu to a user by an AR device consistent with disclosedembodiments. As an example, one or more steps of process 600 may beperformed by AR device 200 shown in FIG. 2. For example, in process 600,a sensor of AR device 200 may detect an orientation of a head of user401 and provide a corresponding output to a processing device of ARdevice 200. The processing device may cause a virtual menu to be shownon the display based on the detected orientation of the head of user401.

At step 610, AR device 200 may monitor an orientation of the head ofuser 401 based on output of at least one sensor associated with the ARdevice. For example, the processing device (e.g., microprocessor 208) ofAR device 200 may execute instructions of positional processing module378, discussed above in relation to FIG. 3, to perform this step. Thesensor may be included in AR device 200 and be configured to provide anoutput indicative of the orientation of the user's head.

At step 620, AR device 200 may determine, based on the monitoredorientation of the head of user 401, whether user 401 is looking upwardor downward with respect to a predetermined horizontal threshold. Forexample, the processing device of AR device 200 may execute instructionsof positional processing module 378 to perform this step. Thepredetermined horizontal threshold may be configured by the positionalprocessing module 378 or preconfigured by the user via a user interface.The predetermined horizontal threshold may be set in units of degrees,radians, or another other units for measurement of angles.

At step 630, AR device 200 may cause a virtual menu to be shown ondisplay 204 if user 401 is determined to be looking upward or downwardwith respect to the predetermined horizontal threshold. For example, theprocessing device of AR device 200 may execute instructions of renderingservices module 370 to perform this step.

The virtual menu displayed in step 630 may be displayed such that theuser perceives the virtual menu as located at a predetermined, fixedangle of rise above the predetermined horizontal threshold. Further, thevirtual menu may be semi-transparent and displayed on the lens. In someembodiments, AR device 200 may change a location at which the virtualmenu is shown on display 204 in response to a sensed change inorientation of the user's head.

The virtual menu displayed in step 630 may include one or moreselectable elements and/or the virtual menu may be a nested menu. Afterreceiving a selection of one or more of the elements, AR device 200 maytake other actions. For example, AR device 204 may determine whetheruser 401 is looking in a direction intersecting any of the selectablemenu items, and cause a selection of a particular item if user 401 looksin a direction of the particular item for a time period that exceeds apredetermined threshold.

Look and Dwell to Select

In some embodiments, a user interface that is displayed by a wearableaugmented reality device may include a menu item that is selectablebased on a user looking at the menu item. The AR device may recognizeselection of the menu item by monitoring a viewing direction of the user(e.g., the orientation of the user's head and/or the orientation of theuser's gaze) to determine the direction in which the user is looking. Inaddition, the AR device may determine whether the user dwells on themenu item longer than a predetermined dwell threshold (e.g., 0.25seconds, 0.5 seconds, one second, three seconds, five seconds, within arange of 0.25 seconds to 2.0 seconds, etc.). If the user dwells on themenu item for the predetermined dwell threshold, the menu item may beselected.

For example, consistent with some embodiments, a wearable deviceprovides a virtual menu to a user. The device comprises a display and atleast one sensor configured to provide an output indicative of a viewingdirection of the user. The device further comprises at least oneprocessing device configured to cause a virtual menu to be shown on thedisplay. The virtual menu includes at least one selectable element. Theat least processing device is further configured to monitor a viewingdirection of the user based on the output of the at least one sensor;determine based on the monitored viewing direction whether the user islooking in a direction of the at least one selectable element of thevirtual menu; determine an amount of time that the user looks in thedirection of the at least one selectable element of the virtual menu;and cause at least action associated with the at least one selectableelement of the virtual menu if the amount of time exceeds apredetermined dwell time threshold.

Referring to FIG. 4, as described above, one or more sensors (e.g., oneor more sensors included in sensor array 125) included in AR device 200may be configured to detect a viewing direction of user 401 (e.g., anorientation of a head of user 401 and/or an orientation of the gaze ofuser 401) and provide an output indicative of the orientation to aprocessing device (e.g., microprocessor 208) of AR device 200. Theprocessing device may be configured to display a virtual menu on display204 in response to actions of user 401. For example, the processingdevice may be configured to display a virtual menu on display 204 if itis determined that the user is looking in a certain predetermineddirection (e.g., to the user's left or right, or upward). In someembodiments, display of the virtual menu may be triggered by one or morehead movements of the user. For example, if the user's head or eyes aredetermined to move in a predetermined pattern (e.g., left-right-left, orany other suitable pattern) or if the user's head or eyes are determinedto move such that the user is determined to be looking in a certainpredetermined direction or orientation, then the processing device maycause the virtual menu to appear on the display. In other embodiments,the menu can be continuously shown on the display, and the processingdevice may cause the menu to be shown outside of a user's primarylooking direction until the user's head movement indicates a desire toaccess the menu. For example, the virtual menu may hover to the left,right, or above a user's primary looking direction, and may be locked inreal world coordinates such that movements of the user's head (e.g., ina manner causing the user's looking direction to intersect with thevirtual location of the menu relative to the real world coordinates) maycause the processing device to move the location of the virtual menurelative to the display (e.g., more toward a center of the user's fieldof view). The virtual menu may include one or more selectable items,such as icons, check-boxes, and/or texts.

As described above in connection with FIG. 4, the processing device ofAR device 200 may be configured to monitor a viewing direction of user401 (e.g., an orientation of the head of user 401 and/or an orientationof the gaze of user 401) based on output received from the one or moresensors. Such sensors may include, for example, one or more componentsassociated with IMU 201 and/or sensor array 125. Such sensors may alsoinclude one or more accelerometers, gyroscopes, magnetometers, eyetracking sensors, etc. as discussed in detail above. For example, thedetected orientation of the head of user 401 may include an angle of theuser's head formed with respect to the horizontal plane, which isassociated with upward or downward movement of the head of the user,along with a direction of the user's head in the horizontal plane, whichmay be associated with left or right movement of the head of the user.For example, the one or more sensors of AR device 200 may outputthree-dimensional coordinates of multiple points of AR device 200 to theprocessing device, and the processing device may determine the angle ofthe user's head with respect to the horizontal plane and the directionof the user's head within the horizontal plane based on the receivedcoordinates.

In some embodiments, the processing device may be configured todetermine the direction of the user's sight line based on the detectedorientation of the user's head and/or the detected orientation of theuser's gaze. For example, the processing device may determine or assumethat the direction of the user's sight line is coincident with a certainhead orientation reference line (e.g., a line extending through andperpendicular to a horizontal and/or vertical midpoint of the display,or any other suitable reference). As another example, the processingdevice may derive the direction of the user's sight line by applying oneor more correction operations relative to a determined head orientationreference. For example, for some users, a sight line may be above,below, leftward, or rightward of a standard head orientation reference,as determined by any combination of the onboard head orientationsensors.

In some embodiments, the processing device of AR device 200 may beconfigured to determine, based on the monitored orientation of the heador the monitored orientation of the user's gaze, whether the user islooking in a direction of a selectable element of the virtual menu. Forexample, positional processing module 378 shown in FIG. 3 may beconfigured to store instructions that, when executed by a processingdevice (e.g., microprocessor 208) of AR device 200, determine whetheruser 401 is looking in a direction of a selectable element of thevirtual menu based on output from the one or more sensors.

For example, the processing device may detect that the direction of theuser's sight line intersects with a particular selectable element of thevirtual menu, and as a result, determine that the user is looking in thedirection of the particular selectable element. In some embodiments, theprocessing device may be configured to divide the area of the virtualmenu shown on display 204 into a number of sections, each of which maycontain one or more selectable menu items. If the direction of theuser's sight intersects with any of the sections, the processing devicemay determine that the user is looking in the direction of the one ormore menu items contained in the section.

In some embodiments, the processing device of AR device 200 may beconfigured to determine an amount of time that user 401 looks in thedirection of particular menu item of the virtual menu. In someembodiments, positional processing module 378 shown in FIG. 3 may beconfigured to store instructions that, when executed by a processingdevice (e.g., microprocessor 208) of AR device 200, determine an amountof time that user 401 looks in the direction of particular menu item ofthe virtual menu.

For example, the sensors of the AR device 200 may periodically detectthe viewing direction of the user (e.g., the orientation of the user'shead and/or the orientation of the user's gaze) at a predetermined timeinterval and provide an output to the processing device. The processingdevice may determine the viewing direction periodically based on theperiodical output received from the sensors. The predetermined timeinterval may be set as every second, multiple times per second (2, 4, 6,10 or more times per second), once over the course of multiple seconds,or any other suitable time interval. This setting may be a defaultsetting or a user adjustable setting.

If the currently detected orientation of the user's viewing directionstays unchanged from the previously detected orientation (e.g., at thelast time interval), the processing device may determine whether user401 has looked in the same direction long enough to trigger an action(e.g., selecting a menu item). However, if the currently detectedorientation of the user's viewing direction has changed from thepreviously detected orientation, the processing device may determinethat user 401 changed the direction of his or her sight line and may setthe amount of elapsed time that the user has looked in the new directionto zero.

In some embodiments, the processing device may allow a small amount ofdifference between the currently detected orientation of the user'sviewing direction and the previously detected orientation and stillconsider the user to have not changed his or her sight line. This mayaccount for slight movements of the user's eyes and/or head such thatthe user does not need to remain perfectly still in order to cause anaction. For example, if the currently detected orientation of the user'sviewing direction changes from the previously detected orientation forless than a predetermined threshold, the processing device may determinethat the direction of user's sight line remains unchanged. On the otherhand, if the currently detected orientation of the user's viewingdirection changes from the previously detected orientation equal to ormore than the predetermined threshold, the processing device maydetermine that the direction of the user's sight line has changed.

If it is determined that user 401 looks in the direction of a particularitem of the virtual menu for an amount of time that exceeds apredetermined dwell time threshold, the processing device of AR device200 may cause an action associated with the particular item to beperformed. For example, if it is determined that user 401 looks in thedirection of a camera icon of the virtual menu for an amount of timethat exceeds the predetermined dwell time threshold, the processingdevice of AR device 200 may activate an onboard camera and/or may causea picture of the user's field of view to be taken (e.g., after apredetermined delay period), among other actions. As another example, ifit is determined that user 401 looks in the direction of a text commandof the virtual menu for an amount of time that exceeds the predetermineddwell time threshold, the processing device of AR device 200 may causean audio output of the text to be provided to the user.

The predetermined dwell time threshold may be set as 0.1 seconds, 0.25seconds, 0.5 seconds, 1 second, or multiple seconds, such as a value of3, 5, or 10 seconds, or any other suitable value. In some embodiments,the dwell time threshold may fall within a range of about 0.1 second toabout 5 seconds. In other embodiments, the dwell time threshold may fallwithout a range of about 0.25 seconds to about 2 seconds. Thepredetermined dwell time threshold may be pre-configured by user 401through a user interface of AR device 200 or be pre-set based on adefault setting of AR device 200. For example, display 204 may display amenu with different values of the predetermined dwell time threshold toenable user 401 to make a selection. As another example, display 204 maydisplay a field that enables user 401 to input a desirable value of thepredetermined dwell time threshold. In some embodiments, display 204 mayalso display options of different actions and allow user 401 toconfigure the specific action to perform in response to looking at thedirection of a menu item for an amount of time that exceeds thepredetermined dwell time threshold. For example, display 204 may displayoptions such as opening an application, providing audio output of text,and/or moving the menu item to the center of the display. Accordingly,user 401 may configure the types of actions that are to be taken by ARdevice 200 when the user looks in the direction of an item for apredetermined period of time.

In some embodiments, as discussed above, alternative or in addition tomonitoring an orientation of the user's head, the processing device ofAR device 200 may be configured to monitor a user's gaze direction basedon output received from one or more eye tracking sensors.

FIG. 7A illustrates an example of a selectable element of virtual menuthat is displayed by an AR device consistent with disclosed embodiments.For example, rendering services module 370, discussed above in referenceto FIG. 3, may store software instructions that, when executed by aprocessing device (e.g., microprocessor 208), cause audio messages to beplayed, and/or cause text messages and/or graphics to be displayed.

As shown in FIG. 7A, a virtual menu 702 which includes multipleselectable items, is displayed on display 204 of AR device 200. Whilethe selectable items of virtual menu 702 in this example include text,they are not limited to text and can be icons, check-boxes, graphics, ora combination thereof. Virtual menu 702 may be semi-transparent suchthat virtual menu 702 is layered on top of the user's field of view anddoes not completely block the user's field of view. Virtual menu 702 mayalso be displayed such that it contrasts with other areas of display204. For example, the color, brightness, and/or transparency of virtualmenu 702 may be different from other areas of display 204. In someembodiments, the transparency of virtual menu 702 may be adjustablebased on input received from user 401. For example, display 204 maydisplay values of transparency associated with virtual menu 702 that areselectable by user 401. As another example, display 204 may display aninput field that enables user 401 to input a desirable transparencyvalue. The transparency value may be set in terms of percentage that theitem is transparent and that light can pass through the item. As anexample, the transparency value may be set as 20, 30, 50 percent or moreaccording to a preference of user 401. In some embodiments, one or morebuttons may be placed on AR device 200 for user 401 to increase ordecrease the transparency of virtual menu 702.

In some embodiments, when user 401 looks at a menu item for a prolongedperiod of time (e.g., an amount of time that exceeds a predetermineddwell time threshold), the processing device of AR device 200 may causean action associated with the menu item to be performed. As illustratedin FIG. 7A, user 401 looks at menu item 703 for a prolonged period oftime. In response, the processing device may cause menu item 703 to behighlighted in display 204 and may initiate the corresponding actionassociated with menu item 703. In this example, menu item 703 defines anoperation to share the user's current field of view with other users.Accordingly, when the processing device determines that the user's sightline intersects with the region of menu item 703 for a prolonged periodof time (e.g., an amount of time that exceeds a predetermined dwell timethreshold), the processing device proceeds to perform the action ofsharing the user's current field of view with other users.

In some embodiments, the processing device of AR device 200 may beconfigured to divide the area of virtual menu 702 into multiplesections, such as sections 704-707 illustrated in FIG. 7A. Each of thesections may include one selectable menu item. The area of each sectionmay be the same or different depending on the size of the menu itemincluded in each section. The area of the sections may be configured tobe slightly larger than the area that the menu item itself occupies suchthat if the direction of the user's sight is sufficiently near the menuitem, even though it does not intersect with the menu item, theprocessing device may still determine that the user intends to invokeactions associated with the menu item to be performed and may proceed toinitiate such actions. If the processing device determines that the useris looking in a direction intersecting the area of any of the sectionsfor a predetermined time period, the processing device may causecorresponding actions associated with the menu item contained within thesection to be performed. For example, if the processing devicedetermines that the user is looking in a direction intersecting the areaof section 706 which contains menu item 703, the processing device maycause corresponding actions associated with menu item 703 to beperformed.

As shown in FIG. 7A, in some embodiments, the processing device maycause menu item 703 to be highlighted on display 204 if it determinesthat the user is looking in a direction intersecting the area of section706 which contains menu item 703. Menu item 703 may remain highlightedduring the time period that the user is looking in a directionintersecting the area of section 706 until the processing deviceinitiates actions associated with menu item 703. In other embodiments,menu item 703 may be highlighted for a predetermined time period, suchas one second or multiple seconds, if the processing device determinesthat the user is looking in a direction intersecting the area of section706 which contains menu item 703. In some embodiments, the processingdevice may cause the entire section 706 to be highlighted if itdetermines that the user is looking in a direction intersecting the areaof section 706 which contains menu item 703.

In some embodiments, if the processing device of AR device 200determines that the user is looking in a direction intersecting theselectable menu item (or the area of the section containing the menuitem) for a predetermined time period (e.g., an amount of time thatexceeds a predetermined dwell time threshold), the processing device maycause an application associated with the menu item to be opened. Forexample, referring to FIG. 7A, the processing device may cause anapplication associated with menu item 703 to be opened, such as a socialmedia application installed on AR device 200 that allows user 401 toshare a digital file with other users. The application may be one of thethird party applications 362 provided in FIG. 3. In some embodiments,the processing device may cause interface of the application associatedwith menu item 703 to be displayed on display 204 such that user 401 mayperform further operations through the interface of the application.

In some embodiments, if the processing device of AR device 200determines that the user is looking in a direction intersecting theselectable menu item (or the area of the section containing the menuitem) for a prolonged time period (e.g., an amount of time that exceedsa predetermined dwell time threshold), the processing device may causean audible output associated with the menu item to be provided. Forexample, referring to FIG. 7A, the processing device may cause anaudible output of the text of menu item 703 to be provided to user 401.The AR device 200 may include a speaker, such as speaker 206 shown inFIG. 2, to provide the audible output of the text to the user. In someimplementations, the speaker may be included in an earbud that isconfigured for placement inside the user's ear (or a bone conductingmicrophone configured for placement in contact with a side of the userhead). In other implementations, the speaker may be placed in a frame ofthe AR device that is configured to secure the AR device to the head ofthe user.

FIG. 7B illustrates another example of a selectable element of virtualmenu that is displayed by an AR device consistent with disclosedembodiments. As shown in FIG. 7B, a virtual menu 708, which includesmultiple selectable items, is displayed on display 204 of AR device 200.The selectable items of virtual menu 708 are presented as graphicalicons. A corresponding operation may be performed in response to aselection of any of the icons. For example, in some embodiments, whenuser 401 looks at one of the icons for a predetermined period of time(e.g., an amount of time that exceeds a predetermined dwell timethreshold), the processing device of AR device 200 may cause an actionassociated with the icon to be performed. As illustrated in FIG. 7B,user 401 looks at icon 709 for a predetermined period of time. Inresponse, the processing device of AR device 200 may cause icon 709 tobe highlighted in the display 204 and initiates the corresponding actionassociated with icon 709. In this example, icon 709 represents anoperation to share the user's current field of view with other users.When the processing device determines that the user's sight lineintersects with the region of icon 709 for a predetermined period oftime (e.g., an amount of time that exceeds a predetermined dwell timethreshold), the processing device may proceed to perform the action ofsharing the user's current field of view with other users.

In some embodiments, the processing device of AR device 200 may beconfigured to divide the area of the virtual menu 708 into multiplesections, such as sections 710-713 illustrated in FIG. 7B. Each of thesections includes one selectable icon. The area of each section may bethe same or different depending on the size of the icon included in eachsection. The area of the sections may be configured to be slightlylarger than the area that the icon itself occupies such that if thedirection of the user's sight line is sufficiently near the icon, eventhough it does not intersect with the icon itself, the processing devicemay still determine that the user intends to invoke actions associatedwith the icon to be performed and may proceed to initiate such actions.If the processing device determines that the user is looking in adirection intersecting the area of any of the sections for a prolongedtime period, the processing device may cause corresponding actionsassociated with the icon contained within the section to be performed.For example, if the processing device determines that the user islooking in a direction intersecting the area of section 712 whichcontains icon 709, the processing device may cause corresponding actionsassociated with icon 709 to be performed.

FIG. 7C illustrates an example of expanding an element of virtual menuthat is displayed by an AR device consistent with disclosed embodiments.In some embodiments, the virtual menu may include a nested menu, and theprocessing device of AR device 200 may be configured to cause a displayof one or more sub-menu elements in response to a user looking in thedirection of a higher level menu element for a predetermined timeperiod. As shown in FIG. 7C, virtual menu 708 is a nested menu. If theprocessing device determines that user 401 is looking at the directionof a higher level menu element for a predetermined period of time, theprocessing device may cause the higher level menu element to beexpanded. As illustrated in FIG. 7C, the processing device determinesthat user 401 is looking at the direction of a rotation icon 716, andconsequently, causes rotation icon 716 to be selected and expanded tosub-menu 714. The sub-menu 714 may be displayed containing differentrotation options, such as clockwise rotation, counter clockwiserotation, rotation lock, etc. The sub-menu 714 may be a text menu, agraphical interface, or a combination of text and graphics.

The sub-menu 714 may include one more selectable sub-menu items. If theprocessing device of AR device 200 determines that user 401 looking inthe direction of a sub-menu item for a predetermined period of time(e.g., any of the times or time ranges associated with the predetermineddwell time threshold), the processing device may cause an actionassociated with the sub-menu item to be performed, such as opening anapplication associated with the sub-menu item, providing an audibleoutput of the sub-menu item, performing a task associated with thesub-menu item, etc. For example, the processing device may cause theclockwise rotation of the current display if user 401 looks in adirection intersecting the clockwise rotation icon 715 in the sub-menufor a time period that exceeds a predetermined threshold. In someembodiments, the processing device may cause a sub-menu item to behighlighted on the display if user 401 looks in a direction intersectingthe sub-menu item. For example, the processing device may cause theclockwise rotation icon 715 to be highlighted on the display if itdetermines that user 401 is looking in a direction intersecting theicon.

FIG. 8 is a flowchart of an exemplary process 800 for causing an actionassociated with an element of a virtual menu to be performed by an ARdevice consistent with disclosed embodiments. As an example, one or moresteps of process 800 may be performed by AR device 200 shown in FIG. 2.For example, in process 800, a sensor of AR device 200 may detect aviewing direction of user 401 (e.g., an orientation of the head of user401 and/or an orientation of the gaze of the eyes of user 401) andprovide a corresponding output to a processing device of AR device 200.The processing device may cause a virtual menu to be shown on thedisplay which includes one or more selectable elements.

At step 810, AR device 200 may cause a virtual menu to be shown ondisplay 204. For example, the processing device of AR device 200 mayexecute instructions of rendering services module 370 to perform thisstep. The virtual menu displayed in step 810 may include one or moreselectable elements and/or the virtual menu may be a nested menu. Theselectable element of the virtual menu may include an icon, a check-box,and/or text. Further, the virtual menu may be semi-transparent anddisplayed on the lens. The transparency of the virtual menu may beadjustable based on an input received by the processing device from user401.

At step 820, AR device 200 may monitor a viewing direction of user 401based on output of at least one sensor associated with the AR device.For example, the processing device (e.g., microprocessor 208) of ARdevice 200 may execute instructions of positional processing module 378,discussed above in relation to FIG. 3, to perform this step. The sensormay be included in AR device 200 and be configured to provide an outputindicative of the viewing direction of the user. For example, the sensormay be configured to provide an output indicative of the viewingdirection of the user by tracking a pattern of movement of anorientation of the head of the user. The sensor may also be configuredto provide an output indicative of the viewing direction of the user bytracking a gaze of the user's eyes.

At step 830, AR device 200 may determine, based on the monitored viewingdirection of user 401, whether user 401 is looking in a direction of aselectable element of the virtual menu. For example, the processingdevice of AR device 200 may execute instructions of positionalprocessing module 378 to perform this step. For example, the processingdevice may determine the direction of the user's sight based on theorientation of the head of user 401 and/or based on the orientation ofthe gaze of the user's eyes. The processing device may further usecoordinates of multiple references points of AR device 200 received fromthe sensors to determine the direction of the user's sight line.

At step 840, AR device 200 may determine an amount of time that the userlooks in the direction of the selectable element of the virtual menu.For example, the processing device of AR device 200 may executeinstructions of positional processing module 378 to perform this step.AR device 200 may determine the amount of time that the user looks inthe direction of the selectable element based on periodical outputreceived from the sensors at a predetermined time interval.

At step 850, AR device 200 may cause at least one action associated withthe selectable element of the virtual menu if the amount of timedetermined in step 840 exceeds a predetermined dwell time threshold. Forexample, the processing device of AR device 200 may execute instructionsof rendering services module 370, command processor 366, and/or thirdparty applications 362 to perform this step. The predetermined dwelltime threshold may be set as 3, 5, 10 seconds, or any other suitabletime value. Further, the predetermined dwell time threshold may beconfigured by the positional processing module 378 or selected by theuser via a user interface.

The at least one action associated with the selectable element mayinclude expanding the selectable element, opening an applicationassociated with the selectable element, providing an audible output ofthe text of the selectable element, performing a command of theselectable element, and so on. AR device 200 may include a speaker andthe audible output of the text of the selectable element may be providedto the user via the speaker. For example, the speaker may be included inan earbud configured for placement inside the user's ear.

Nested Menu

Some embodiments may include a nested menu system that enables the userto look toward a menu shown on a display, select the menu, and expandthe menu upon selection. The menu may appear to the user at a fixedlocation relative to real world coordinates. In some embodiments, thedevice may use eye tracking to control scrolling through a menu item.

For example, consistent with an embodiment, a wearable device provides avirtual menu to a user. The device comprises a display and at least onesensor configured to provide an output indicative of a viewing directionof a user (e.g., an orientation of a head of the user and/or anorientation of a gaze of the user's eyes). The device further comprisesat least one processing device configured to monitor a viewing directionof the user based on the output of the at least one sensor. The at leastone processing device is further configured to provide a menu on thedisplay and determine based on the monitored viewing direction whetherthe user is looking in a direction of the location of the menu on thedisplay. The at least one processing device is further configured toexpand the menu if the user is determined to be looking in the directionof the location of the menu on the display.

Referring to FIG. 4, as described above, one or more sensors (e.g., oneor more sensors included in sensor array 125) included in AR device 200may be configured to detect a viewing direction of user 401 and providean output indicative of the viewing direction to a processing device(e.g., microprocessor 208) of AR device 200. The processing device maybe configured to display a virtual menu on display 204 in response toactions of user 401. For example, the processing device may beconfigured to display a virtual menu on display 204 if it is determinedthat the user is looking upward with respect to a predeterminedhorizontal threshold, to the right or to the left by a certain amount,or based on any other suitable trigger for displaying the menu. In someembodiments, the menu may be displayed based on voice commands from theuser or by touch input to one or more user interface componentsassociated with AR device 200. In some embodiments, the virtual menu mayinclude a nested menu that can be expanded to display one or more newmenu items not shown on the initial display of the menu.

As described above in connection with FIG. 4, the processing device ofAR device 200 may be configured to a viewing direction of user 401 basedon output received from the one or more sensors (e.g., any combinationof accelerometers, gyroscopes, magnetometers, eye tracking sensors, orother suitable sensors, as discussed in more detail above). Theprocessing device may be configured to determine whether user 401 islooking in a direction of the location of the menu on the display basedon the monitored viewing direction of user 401. In some embodiments,positional processing module 378, as shown in FIG. 3, may be configuredto store instructions that, when executed by a processing device (e.g.,microprocessor 208) of AR device 200, determine whether the user islooking in a direction of the location of the menu based on output fromthe one or more sensors.

For example, the processing device may determine the direction of theuser's sight line based on the monitored viewing direction of user 401.The processing device may identify a region on display 204 that includesthe virtual menu. The identified region may be the same as or slightlylarger than the area the virtual menu occupies. If the direction of theuser's sight falls into the identified region, the processing device maydetermine that user 401 is looking in the direction of the location ofthe menu. On the other hand, if the direction of the user's sight fallsoutside the identified region, the processing device may determine thatuser 401 is not looking in the direction of the location of the menu.

In some embodiments, the processing device may be configured todetermine the time duration that user 401 looks in a direction of thelocation of the menu based on output received from the sensor. Forexample, the sensors of AR device 200 may sense the viewing direction ofthe user at a predetermined time interval (e.g., 2, 4, 6, 10 or moretimes per second) and provide an output to the processing device. Theprocessing device may determine the time duration that user 401 looks ina direction of the location of the menu based on whether the sensedorientation of the user's head changes from the previously detectedorientation. For example, if the currently detected viewing direction ofthe user stays unchanged from the previously detected viewing direction(e.g., at the last time interval), the processing device may increasethe amount of time that the user looks in the location of the menu bythe predetermined time interval. If the currently detected viewingdirection changes from the previously detected viewing direction, theprocessing device may determine that the user moves the direction of hisor her sight line away from the location of the menu and may reset theamount of elapsed time that the user has looked in the location of themenu to zero. If the time duration that the user 401 looks in adirection of the location of the menu is less than a predetermined timethreshold, the processing device may determine that user 401 does notintend to access the nested menu and may not take any action. On theother hand, if the time duration that the head of user 401 stays in adetected viewing direction is greater than or equal to the predeterminedtime threshold, the processing device may determine that user 401intends to access the nested menu and may cause the menu to be expandedon display 204.

In some embodiments, the processing device may allow a small amount ofdifference between the currently detected viewing direction of the userand the previously detected viewing direction and still consider theuser to have not changed his or her sight line. This may account forslight movements of the user's head and/or eyes such that the user doesnot need to remain perfectly still in order to cause a nested menu to beexpanded. For example, if the currently detected viewing direction ofthe user changes from the previously detected orientation for less thana predetermined orientation threshold, the processing device maydetermine that the user is looking into the direction of the location ofthe nested menu and cause the menu to be expanded. On the other hand, ifthe detected viewing direction of the user changes from the previouslydetected orientation for equal to or more than the predeterminedorientation threshold, the processing device may determine that the useris not looking into the direction of the location of the nested menuanymore and may not cause the menu to be expanded.

In some embodiments, a discussed above, alternative or in addition tomonitoring an orientation of the user's head, the processing device ofAR device 200 may be configured to monitor a user's gaze direction basedon output received from one or more eye tracking sensors. For example,as discussed above, the processing device may be configured to determinewhether user 401 is looking in a direction of the location of the menuon the display based on the direction of the user's gaze.

FIG. 9A illustrates an example of a nested menu that may be displayed byan AR device by consistent with disclosed embodiments. For example,rendering services module 370, discussed above in reference to FIG. 3,may store software instructions that, when executed by a processingdevice (e.g., microprocessor 208), cause a nested menu including textand/or graphics to be displayed. Rendering services module 370 mayfurther store instructions that are executable by the processing deviceto cause the nested menu to be expanded into one or more sub-menus inresponse to user input.

As shown in FIG. 9A, a nested menu 902 is displayed on display 204 of ARdevice 200. Nested menu 902 may include text, icons, check-boxes,graphics, or a combination thereof. In some embodiments, user 401 maylook in the direction of the location of nested menu 902 in order toaccess the nested menu. The processing device of AR device 200 maydetermine that user 401 is looking in the direction of the location ofnested menu 902 based on the monitored orientation of the head of user401 and/or based on the monitored orientation of the user's gaze. Inresponse to the determination, the processing device may cause nestedmenu 902 to be expanded into sub-menu 903 on display 204. As illustratedin FIG. 9A, sub-menu 903 may include new menu items that were notpreviously shown on nested menu 902. Although the items that are shownon sub-menu 903 include text in this example, the sub-menu items are notlimited to text and can be text, icons, check-boxes, graphics, or acombination thereof. Sub-menu 903 may be displayed near nested menu 902such that user 401 does not need to move the direction of the user'ssight drastically in order to access sub-menu 903.

In some embodiments, sub-menu 903 may be displayed at a predeterminedlocation on display 204, such as the center of display 204, an upperportion of display 204, or any other suitable location. Thepredetermined location for sub-menu 903 to be displayed may bepre-configured by user 401 through a user interface of AR device 200.For example, display 204 may allow user 401 to select a location ofdisplay 204 at which sub-menu 903 should display. In some embodiments,after sub-menu 903 is displayed at a pre-set location, user 401 may beallowed to drag sub-menu 903 to another location on display 204 that ispreferred by user 401. The processing device of AR device 200 may setthe new location as a default location for displaying sub-menu 903, anddisplay sub-menu 903 at the new location subsequently when theprocessing device determines to expand nested menu 902 in futureinstances. In other embodiments, the location of sub-menu 903 may belocked to a predetermined position of display 204 such that user 401cannot reposition the sub-menu.

In some embodiments, the processing device of AR device 200 may causenested menu 902 to be highlighted on display 204 if the processingdevice detects that the user is looking in the direction of the locationof nested menu 902. Nested menu 902 may remain highlighted during thetime period that the user is looking in the direction of the location ofnested menu 902 until sub-menu 903 is displayed on display 204. If thedirection of the user's sight line is moved away from the location ofnested menu 902 before a sub-menu is being displayed, the processingdevice may stop highlighting nested menu 902 and not display thesub-menu to the user. In other embodiments, nested menu 902 may behighlighted for a predetermined time period, such as one second ormultiple seconds, if the processing device determines that user 401 islooking in the direction of the location of nested menu 902.

Once a nested menu is provided on display 204, AR device 200 may beconfigured to display the nested menu in a fixed location relative tothe display, such as in a locked mode. For example, nested menu 902 maybe locked in the user's field of view while user 401 moves his or herhead to a new position. That is, once nested menu 902 has been providedon the display, nested menu 902 may be locked to a particular locationrelative to display 204 such that subsequent movements of the head ofthe user do not cause movement of nested menu 902 relative to display204.

In other embodiments, nested menu 902 may be provided at a fixed,perceived location relative to the user, such as in an unlocked mode.For example, nested menu 902 may appear on the display whenever the headof user 401 is moved to a particular orientation, e.g., rotated acertain degree to the right, to the left, or vertically. Once the nestedmenu is displayed in such an unlocked mode, the location of the nestedmenu may be changed as the user rotates the head toward the perceivedlocation of the menu (as described in more detail above). For example,once nested menu 902 is shown on the display, subsequent head movementsand/or eye movements further in the upward direction may cause thelocation of nested menu 902 to move downward on the display. Similarly,subsequent head movements and/or eye movements in a downward directionmay cause the location of nested menu 902 to move upward on the display.Additionally, the processing device may be configured to also causechanges in the left-right position of the nested menu relative to thedisplay in response to detected horizontal movements of the user's headand/or eyes (e.g., right-left changes in the looking direction of theuser's head and/or eyes).

In other embodiments, the nested menu may be provided at a fixedlocation relative to real world coordinates, where the fixed locationremains constant irrespective of user movements. In such embodiments,display of the nested menu may depend on whether the user's headorientation and/or gaze direction of the user's eyes are determined, bythe processing device, to cause the fixed, virtual position of the menuto be within the user's field of view. When the menu's location isdetermined to be within the user's field of view, it is provided to thedisplay. When the menu's location is determined to be outside of theuser's field of view, the menu is not displayed. For example, the usermay perceive that the nested menu occurs at a particular location ofparticular coordinates in the real world, and when the user moves orlooks away from that particular location, the nested menu may not appearon the display. That is, the display of the nested menu may depend notonly on the specific viewing direction of the user, but also thephysical location/orientation of the user or user's head and/or eyesrelative to real world coordinates. Even if the orientation of the headand/or eyes of the user remains the same, if the physical location ofthe user changes, the nested menu may disappear from the display. Thenested menu may reappear on the display when the user looks again at theparticular location in the real world designated as the perceivedlocation of the menu in real world coordinates.

The above described methods of moving nested menu 902 on the display canbe applied similarly to sub-menu 903, items shown on sub-menu 903, andthe like. It should be understood that one method may be applied to movethe location of nested menu 902 on the display, such as the locked mode,while another method may be applied to move the location of sub-menu903, such as the unlocked mode.

In some embodiments, sub-menu 903 may contain multiple items, and it maynot be desirable to display the complete list of menu items on display204. The processing device of AR device 200 may be configured to show apartial list of menu items when nested menu 902 is expanded intosub-menu 903. For example, the processing device may display apunctuation mark, such as an ellipsis, on the bottom of sub-menu 903,indicating to the user that additional menu items are available and notbeing currently displayed. As illustrated in FIG. 9A, sub-menu 903 showsthree menu items and an ellipsis indicating that more menu items areavailable but are not currently in display. In some implementations, theprocessing device may be configured to place the most frequently usedmenu items by user 401 on the top list of sub-menu 903 such that thoseitems may be shown to user 401 when nested menu 902 is expanded intosub-menu 903. In other implementations, the processing device may beconfigured to allow user 401 to make a selection via a user interface asto the order of the menu items being shown on the display.

FIG. 9B illustrates another example of a nested menu that is displayedby an AR device consistent with disclosed embodiments. In someembodiments, the processing device of AR device 200 may be configured toscroll through a virtual menu in response to user input. The user inputmay include an output of an eye tracking sensor configured to determinea direction of the user's gaze. For example, after sub-menu 903 isdisplayed, user 401 may look in a direction of the ellipsis at thebottom of sub-menu 903, or downward relative to the location of sub-menu903, continuously for a certain period of time, such as one second ormultiple seconds. In response to the user looking in the direction ofthe ellipsis at the bottom of sub-menu 903 or a downward directionrelative to the location of sub-menu 903, the processing device may beconfigured to scroll down sub-menu 903. As shown in FIG. 9B, in sub-menu904, the menu items shown on sub-menu 903 are scrolled down for oneline, the menu item “zoom in” on the top line of sub-menu 903 is notshown in sub-menu 904, and a new menu item 905 is shown in sub-menu 904.If user 401 changes the direction of sight to the ellipsis at the top ofsub-menu 904, or an upward direction relative to the location ofsub-menu 904, continuously for a certain period of time, the processingdevice may be configured to scroll up sub-menu 904, and sub-menu 903 maybe displayed again on the display.

In some embodiments, user 401 may be able to configure the speed themenu is scrolled through. For example, AR device 200 may display aninterface for user 401 to configure the time period that the menu willscroll through one line. For example, user 401 may select 0.5, 1, or 2seconds, or any other suitable time interval that before a menu toscroll up or down for a single line. Thus, if user 401 looks at adirection downward to the location of sub-menu 903 for 2 seconds,sub-menu 903 may be scrolled down for 4 lines, 2 lines, or 1 linedepending on the user's selection of the time interval for the menu tobe scrolled through for a single line.

In some embodiments, the processing device of AR device 200 may beconfigured to display a scrolling mark on the sub-menu, such as anellipsis, an arrow, or a scrolling bar. If it is determined that theuser is looking in a direction of the scrolling mark, the processingdevice may cause the sub-menu to be scrolled up or down in thecorresponding direction. For example, if the processing devicedetermines that the user is looking in a direction of a scrolling markplaced in the upper region of the sub-menu, the processing device maycause the sub-menu to be scrolled up. If the processing devicedetermines that the user is looking in a direction of a scrolling markplaced in the lower region of the sub-menu, the processing device maycause the sub-menu to be scrolled down.

In some embodiments, the processing device of AR device 200 may beconfigured to identify a region above or below the menu such that if itis determined that the user's gaze falls into the region, the processingdevice may cause the menu to be scrolled up or down. For example, theprocessing device may identify a rectangular region on display 204 thatis located directly above sub-menu 904 such that if it is determinedthat a direction of the user's gaze falls into the region, theprocessing device may cause sub-menu 904 to be scrolled up. Similarly,the processing device may identify a rectangular region on display 204that is located directly below sub-menu 904 such that if it isdetermined that a direction of the user's gaze falls into the region,the processing device may cause sub-menu 904 to be scrolled down.

FIG. 10 illustrates an example of accessing a nested menu that isdisplayed by an AR device consistent with disclosed embodiments. Asshown in FIG. 10, a reticle 1001 may be shown on display 204. Reticle1001 may constitute a virtual reticle shown on display 204 whoseposition on display 204 may be changed in response to user input. Forexample, one or more eye tracking sensors, as described above, mayenable tracking of a user's gaze direction, and the position of reticle1001 on display 204 may be changed with determined changes in the user'sgaze direction.

In some embodiments, reticle 1001 may be fixed relative to display 204(e.g., may be physical inscribed or otherwise marked on display 204 orany other suitable component of AR device 200) such that reticle 1001moves with changes in the user's head orientation. In such embodiments,reticle 1001 may be used, e.g., to selectively overlap with and selectportions of a nested menu that remains unlocked relative to display 204and fixed relative to real world coordinates.

In some embodiments, reticle 1001 may be turned on or off on display 204by user input. For example, if user 401 presses the portion of display204 at which reticle 1001 is shown for a certain period of time, theprocessing device may be configured to remove reticle 1001 from display204. As another example, one or more buttons may be placed on AR device200 for user 401 to turn on or turn off reticle 1001 on display 204.Display of reticle 1001 and/or selection of any menu components may alsobe facilitated by recognized voice commands or recognized hand gestures(or any other suitable triggers) made by the user.

In some embodiments, reticle 1001 may be shown at the center of display204 when it is initially turned on, and may be moved to other locationsin response to user input. For example, user 401 may drag reticle 1001from one position to another position on display 204 by touching display204. Alternatively or additionally, the processing device may beconfigured to move the position of reticle 1001 in response to sensedchanges in the user's gaze direction (as discussed above). Further, asdiscussed above, the position of reticle 1001 relative to any displayedmenu items may be changed by tracking the user's head orientation andupdating the position on display 204 at which the menu items aredisplayed as the user's head orientation changes. The processing devicemay move the position of the menu items in the opposite direction as thedetected direction of the movement of the head of user 401, such thatreticle 1001 intercepts any of the displayed menu items. Regardless ofthe configuration of reticle 1001 (e.g., whether fixed relative todisplay 204 or floating relative to display 204), if the processingdevice determines that the position of reticle 1001 is moved tointercept a portion of the nested menu 1002, the processing device mayexpand the nested menu 1002 to sub-menu 1003 or make a selection of anyportion of nested menu 1002 or sub-menu 1003.

As shown in FIG. 10, if the processing device determines that theposition of reticle 1001 is moved to intercept a portion of nested menu1002, the processing device may expand the nested menu 1002 to sub-menu1003. The reticle 1001 may intercept any part of nested menu 1002, suchas upper part, lower part, or other parts of the menu, to cause it to beexpanded. In some embodiments, the processing device may cause nestedmenu 1002 to be expanded if the area of the portion of the menu that isintercepted by reticle 1001 is greater than or equal to a predeterminedarea threshold. If the area of the portion of the menu that isintercepted by reticle 1001 is less than the predetermined areathreshold, the processing device may determine that the user does notintend to expand the nested menu and would not cause the nested menu tobe expanded. The predetermined area threshold may be set as a percentageof the area of the nested menu, such as 10 percent, 20 percent, or moreof the area of the nested menu.

After nested menu 1002 is expanded into sub-menu 1003, user 401 mayfurther move the position of reticle 1001 to the position of certainitem of sub-menu 1003 to cause further actions to be performed by ARdevice 200. For example, user 401 may move the position of reticle 1001to intercept the sub-menu item “share” on sub-menu 1003, andcorrespondingly, the processing device may proceed to perform the actionof sharing the user's current field of view with other users.

FIG. 11 is a flowchart of an exemplary process 1100 for accessing anested menu by an AR device consistent with disclosed embodiments. As anexample, one or more steps of process 1100 may be performed by AR device200 shown in FIG. 2. For example, in process 1100, a sensor of AR device200 may detect an orientation of a head of user 401 and/or anorientation of the gaze of the eyes of user 401 and provide acorresponding output to a processing device of AR device 200.

At step 1110, AR device 200 may monitor a viewing direction of the user(e.g., an orientation of the head of the user and/or an orientation ofthe gaze direction of the user's eyes) based on the output of the atleast one sensor associated with the AR device. For example, theprocessing device (e.g., microprocessor 208) of AR device 200 mayexecute instructions of positional processing module 378, discussedabove in relation to FIG. 3, to perform this step. The sensor may beincluded in AR device 200 and be configured to provide an outputindicative of the orientation of the user's head. For example, thesensor may be configured to provide an output indicative of the viewingdirection of the user by tracking a pattern of movement of anorientation of the head of the user. As another example, the sensor maybe configured to provide an output indicative of the viewing directionof the user by tracking a gaze of the user's eyes.

At step 1120, AR device 200 may provide a menu on display 204. Forexample, the processing device of AR device 200 may execute instructionsof rendering services module 370 to perform this step. The processingdevice may show the menu on display 204 such that the user perceives themenu location as fixed relative to real world coordinates and variablerelative to display coordinates. In some embodiments, the processingdevice may be configured to change a location of the menu relative tothe display when the user rotates the head toward the menu.

At step 1130, AR device 200 may determine, based on the monitoredviewing direction of user 401, whether user 401 is looking in adirection of the location of the menu on the display. For example, theprocessing device of AR device 200 may execute instructions ofpositional processing module 378 to perform this step. For example, theprocessing device may determine the direction of the user's sight linebased on the orientation of the head of user 401 and/or the orientationof the gaze direction of the eyes of user 401. The processing device mayfurther use coordinates of multiple references points of AR device 200received from the sensors to determine the direction of the user's sightline.

At step 1140, AR device 200 may expand the menu if the user isdetermined to be looking in the direction of the location of the menu onthe display. For example, the processing device of AR device 200 mayexecute instructions of rendering services module 370 to perform thisstep. Expanding the menu may include showing to the user at least onemenu element not shown upon initial display of the menu. Expanding themenu may also include exploding the menu into a plurality of new menuitems shown on the display, and the plurality of new menu items may bearranged around an originally displayed menu item. In some embodiments,the processing device may be configured to display a reticle on thedisplay and expand the menu if the user moves a position of the reticleto intercept at least a portion of the menu.

Look-Initiated Communication

In some embodiments, a wearable AR device may be configured to initiateand maintain a communication link with another device, such as anotherAR device, a smartphone, a tablet, or other computing device. That is,in one example, a wearable AR device may initiate a communication linkthat allows a user of the device to communicate with another individual,such as another user of a wearable AR device. The communication link mayinclude any electronic pathway through which two or more individuals maycommunicate (e.g., by speaking, sharing information, etc.) with eachother.

A wearable AR device that is configured to allow a user to communicatewith another individual presents an opportunity for an enhanced and/orimproved communication experience. For example, such AR devices mayallow individuals to communicate with each other in situations in whichalternative forms of communication may be difficult or impossible. Suchcommunication may include, for example, private or secretivecommunication, remote communication, communication in environments thatare too loud or too quiet for speaking at natural volume, etc. Further,such a device may enhance convenience and reduce effort when it comes tocommunicating (e.g., directly speaking) with another person.

A wearable AR device according to the present disclosure may includefeatures that allow for initiation of a communication link through aprocess that incorporates aspects of natural communication experiencesand minimizes the steps necessary to open the communication link. Thesefeatures may enhance the user's augmented reality experience, allowingfor increased and efficient communication abilities between individualswhile providing a natural and easy-to-use experience.

In an exemplary embodiment, a wearable AR device may initiate acommunication link based on an orientation of the device. For example,in an embodiment in which a wearable AR device is worn on a head of auser, the device may initiate a communication link based on a headorientation of the user. In one example, a wearable AR device may beconfigured for communication by a user orientating a head-worn AR devicetowards a person of interest, resulting in the wearable AR deviceopening a communication link between the user and the person ofinterest. In some embodiments, alternative or in addition to monitoringan orientation of the user's head, the processing device of AR device200 may be configured to monitor a user's gaze direction based on outputreceived from the one or more eye tracking sensors. For example, theprocessing device may be configured to initiate a communication linkbased on the direction of the user's gaze.

FIG. 12 illustrates an example of a user 1201 wearing an AR deviceconsistent with disclosed embodiments. As shown in FIG. 12, user 1201 isin an environment 1200 and is wearing AR device 200, as discussed aboveregarding FIG. 2. AR device 200 may include a frame configured to securethe device to the head of user 1201. As discussed above, AR device 200may include sensor array 125. Sensor array 125 may include one or moresensors configured to detect the viewing direction of user 1201 bydetecting the position and/or orientation of AR device 200. The one ormore sensors may include any one or more of a motion detector (e.g., agyroscope and/or an accelerometer), a camera, a location determinationdevice (e.g., a GPS device and/or an NFC reader), a magnetometer, and/oran orientation sensor (e.g., a theodolite, infra-red sensor). Suchmagnetometers (compasses) may detect orientation or a direction that auser is looking, accelerometers (e.g., three-axis accelerometers) todetect the direction and speed of movements associated with AR device200, or the like. In addition, as discussed above, AR device 200 mayinclude input/output features that enable communication with anotherdevice, such as a microphone, audio buds/speakers, a front facingcamera, and a display. Further, as discussed above, sensor array 125 mayinclude one or more eye tracking sensors configured to detect a viewingdirection of the user's gaze.

In some embodiments, a memory of AR device 200 (e.g., positionalprocessing module 378) may be configured to store instructions that whenexecuted by a processing device (e.g., microprocessor 208) of AR device200, determine the viewing direction of user 1201 (e.g., the orientationof the head of user 1201 and/or the gaze direction of the eyes of user1201) based on output from the one or more sensors. The processingdevice may be further configured to execute instructions to initiate acommunication link between AR device 200 and another device (e.g.,another AR device), based on the determined viewing direction of user1201.

In one embodiment, AR device 200 may be oriented toward a direction thatcorresponds to a user view v1. For example, in an embodiment in which ARdevice 200 is wearable as glasses, AR device 200 may be oriented in adirection that user 1201 is looking. In this way, a head orientationdetermined by the sensors associated with AR device 200 may correspondto a direction that user 1201 is looking.

In an exemplary embodiment, AR device 200 may be configured to initiatea communication link with a device 200A based on a determination that ARdevice 200 is oriented towards a device 200A and/or a user associatedwith device 200A. For example, as shown in FIG. 12 user 1201 may look ata user 1202 (i.e., user 1202 is located in the user view v1), who maypossess (e.g., be wearing) device 200A, which in an exemplary embodimentis another AR device. It should be understood that device 200A may be ofthe same or a different type than AR device 200 being used by user 1201,and is not necessarily an augmented reality device. That is, user 1202may possess and/or wear any device that may provide its location toserver system 110 (e.g., via a smartphone, tablet, computer, etc., usingGPS) and/or that may have communication capabilities. For example, user1202 may be using a smartphone and user 1201 may identify a location ofand/or communicate with the smartphone using AR device 200.

Based on at least the determination that AR device 200 is orientedtoward user 1202 (and/or device 200A), AR device 200 may be configuredto initiate a communication link 1203 that allows user 1201 tocommunicate with user 1202 and, in some embodiments, vice versa. Forexample, in some embodiments, as user 1201's AR device 200 rotates inthe environment and its field of view changes, different people ofinterest and/or representations (e.g., an image, text, icon, etc.) ofpeople of interest may be displayed on the display of AR device 200 whenits field of view overlaps with the location of each person of interest.Then looking at a person of interest or an icon representing them mayestablish a communication link, even if the person is using a differenttype of device (e.g., a smartphone).

AR device 200 may be configured to identify people of interest anddisplay them and/or representations of them based on informationreceived by AR device 200. For example, AR device 200 may populate alist of people of interest from a list of contacts received by AR device200. In some embodiments, AR device 200 may designate any individualpossessing a device, to which AR device 200 may establish acommunication link, as a person of interest. It should be understoodthat AR device 200 my define people of interest in advance of user 1201using AR device 200. In some embodiments, server system 110 mayautomatically define people of interest. For example, certain groups(e.g., corporations, military units, etc.) may register associatedindividuals with server system 110 and allow any AR device connected toserver system 110 to use the registered individuals as a list of peopleof interest. AR device 200 may receive location information associatedwith people of interest and display representations of these people touser 1201 (e.g., when they are within a certain distance of AR device200).

As shown in FIG. 12, communication link 1203 may be a pathway throughwhich user 1201 may communicate with user 1203. For example,communication link 1203 may allow user 1201 to speak into AR device 200such that the speech is reproduced by device 200A. For instance, usermay speak into a microphone of AR device 200, which may transmit audiodata to device 200A over a network (e.g., cellular connection, Wi-Ficonnection, radio connection, etc.). Device 200A may interpret the audiodata and audibly provide user 1201's voice to user 1202, such as throughan associated audio bud/speaker. In some embodiments, user 1202 maycommunicate with user 1201 in a similar manner (e.g., by speaking into amicrophone associated with device 200A). In this way, communication link1203 may allow user 1201 to communicate with user 1202 based on theorientation of AR device 200.

FIG. 13 depicts another embodiment, in which a user 1301 in anenvironment 1300 may communicate with a group of users 1302, 1303, and1304. In an exemplary embodiment, user 1301 may wear AR device 200. ARdevice 200 may be configured to initiate a communication link 1305 basedon a head orientation of AR device 200, in manner similar to thatdescribed above with respect to FIG. 12. For example, AR device 200 maybe configured to initiate communication link 1305 between AR device 200and a device 200A possessed (e.g., worn) by user 1302 based on adetermination that user 1301 is looking at user 1302. Users 1301 and1302 may subsequently communicate with (e.g., speak to) each otherthrough communication link 1305, using AR device 200 and device 200A. Insome embodiments, as discussed above, alternative or in addition tomonitoring an orientation of the user's head, the processing device ofAR device 200 may be configured to monitor a user's gaze direction basedon output received from the one or more eye tracking sensors. Forexample, the processing device may be configured to initiate acommunication link 1305 based on the detected gaze direction of user1301.

In the embodiment of FIG. 13, communication link 1305 may further allowuser 1301 to communicate with users 1303 and 1304, who may possess(e.g., wear) devices of their own, such as AR devices 200B and 200C,respectively. For example, AR device 200 may determine that users 1302,1303, and 1304 are part of a group, and therefore communication with anyone of devices of 200A, 200B, and 200C may be replicated across all ofthe devices such that user 1301 may communicate (e.g., by speaking,sharing information, etc.) with each of users 1302, 1303, and 1304.Users 1302, 1303, and 1304 may communicate with each other and/or user1301 in a similar manner. Further, while AR device 200 is described asinitiating communication link 1305 based on being oriented toward user1301, it should be understood that communication link 1305 may beinitiated based on AR device 200 being oriented toward any of users1302, 1303, 1304 (or being generally oriented toward two or more ofthem).

It should be understood that communication link 1305 may be configuredsuch that various types of group communication may be possible. In someinstances, communication link 1305 may operate like a conference call,where all individuals connected through communication link 1305 can hearand can talk to all others. In other instances, communication link 1305may be one way communication where members of the group (e.g., users1302, 1303, 1304) can hear a communication leader (e.g., user 1301) andcan talk with the communication leader but cannot hear or talk withothers in the group. AR device 200 may be configured such that a usercan select the type of group communication to be used.

FIG. 14 is a flowchart of an exemplary process 1400 for providing acommunication link between at least two users, such as users 1201 and1202 of FIG. 12, wearing AR device 200 and device 200A, respectively. Inone example, one or more steps of process 1400 may be performed by an ARdevice 200 as shown in FIG. 2. For example, user 1201 may wear AR device200 and move their head and/or eyes in a particular viewing direction.The sensors associated with AR device 200 may determine the viewingdirection of user 1201.

At step 1410, AR device 200 may monitor a viewing direction user 1201(e.g., an orientation of the head of user 1201 and/or a gaze directionof the eyes of user 1201) based on output of at least one sensorassociated with AR device 200. For example, a processing device (e.g.,microprocessor 208) of AR device 200 may execute instructions ofpositional processing module 378, discussed above in relation to FIG. 3,to perform this step. The sensor may be included in AR device 200 andmay be configured to provide an output indicative of the viewingdirection of the user. For example, the sensor may be configured toprovide an output indicative of the viewing direction of the user bytracking a pattern of movement of an orientation of the head of theuser. As another example, the sensor may be configured to provide anoutput indicative of the viewing direction of the user by tracking agaze of the user's eyes.

At step 1420, AR device 200 may determine whether the user is looking ina direction of an individual. For example, AR device 200 may compare aviewing direction of user 1201 with additional information to determinethat user 1201 is looking in a direction of user 1202. The additionalinformation may include, for example, a location of user 1202. Thelocation of user 1202 may be determined, for example, through input froma sensor (e.g., a camera) associated with AR device 200 and/or throughlocation data determined by device 200A and sent to AR device 200 (e.g.,over a wireless network). Further, as discussed above, AR device 200 maylocate any suitable device (e.g., a smartphone, tablet, computer, etc.)and provide information to AR device 200 regarding devices that areavailable for communication.

At step 1430, AR device 200 may establish a communication link betweenthe user and the individual. For example, AR device 200 may establish acommunication link 1203 between user 1201 and user 1202, via AR device200 and device 200A. The established communication link may be a pathwaythrough with user 1201 may communicate (e.g., speaking, sharinginformation, etc.) with user 1202. As described herein, thecommunication link may be two-way (e.g., user 1202 can reciprocallycommunicate with user 1201), although not necessarily.

In some embodiments, the communication link may be established when user1201 speaks into AR device 200. For example, AR device 200 may determinethat the speech is intended for user 1202 (since user 1201 is looking atuser 1202) and transmit audio data to device 200A. In some embodiments,AR device 200 may determine whether user 1202 is oriented toward user1201 before a communication link is established. It should beunderstood, however, that, in at least some embodiments, user 1202 neednot be looking at user 1201 in order for AR device 200 to determine thatuser 1201 is looking in a direction of an individual (and/or establish acommunication link). Similarly, AR device 200 may determine whetheranother individual is a recognized person. For example, AR device 200may process image data from a sensor (e.g., camera) and determine thatat least a portion of an individual is in the image data. AR device 200may compare image data with stored image data, to determine whether theat least one individual is a recognized person. In some embodiments, ARdevice 200 may establish a communication path between the user and theat least one individual if the at least one individual is a recognizedperson.

Once the communication link is established, users 1201 and 1202 maycontinue to communicate with each other until the communication link isdeactivated. In an exemplary embodiment, the communication link mayremain active even if user 1201 stops looking at user 1202 (e.g., movestheir head around). AR device 200 may include a feature to allow thecommunication link to be deactivated. For example, AR device 200 maydeactivate the communication link when AR device 200 determines thatuser 1201 is no longer looking at user 1202 (e.g., for a thresholdperiod of time). In other embodiments, user 1201 may cause thecommunication link to be deactivated by pressing a button, using a voicecommand, looking in a certain direction (e.g., up) or in a certainpattern, selecting a hang up icon on the display with eye tracking or bymoving a reticle to a hang up icon.

It should be understood that AR device 200 may perform a process similarto process 1400 to communicate with a group of individuals. For example,as described with respect to FIG. 13, AR device 200 may establish acommunication link between users and determine that the communicationlink should be used to connect to additional users, such as otherindividuals in the area and possessing (e.g., wearing) their own devicescapable of communication through the communication link.

In some embodiments, AR device 200 may be configured to establish acommunication link between a user and another individual who isphysically separated from the user by a wall, building, and/orgeographic distance which would prevent the user and the otherindividual from seeing each other. For example, AR device 200 maypresent an indication (e.g., text, an icon, an image of a person's face,etc.) in the view of the user through a display associated with ARdevice 200. The indication may identify other individuals that the useris facing, even though the use cannot see them. AR device 200 maydetermine the existence of these individuals based on an orientation ofAR device 200 and location information associated with the otherindividuals.

As described above, AR device 200 may display representations ofindividuals in a manner consistent with their relative location withrespect to AR device 200. For example, in some embodiments, arepresentation (e.g., an image, text, icon, etc.) representing a personof interest may be displayed in a user's field of view, if that user'slocation is or would be in the field of view. A user may select therepresentation to establish a communication link.

The user may select the representation by, for example, orienting areticle with the representation. For example, AR device 200 may displaya reticle on its display and allow head-movements made by the user toindicate the direction or location of the representation to be selectedby lining up the reticle with the representation. In some embodiments, aparticular representation may be selected if the time that the userlooks in the direction of the representation on the display exceeds apredetermined dwell time threshold (e.g., 1 second, 2 seconds, 3seconds, 5 seconds, 10 seconds, etc.). It should be understood, however,that other selection mechanisms are possible (e.g., pressing a button,speaking an audible command, etc.). After the user has selected therepresentation, a communication link may be established between theuser's AR device and the other user's device.

In the above embodiment, by rotating or changing head orientation and/oreye orientation and, therefore, the associated field of view, a user maysee the individuals that are located in a position that is or would bein the user's field of view (e.g., are located in a position that isgenerally aligned with a direction that the user is looking). Further itshould be understood that AR device 200 may present informationassociated with the location of individuals in a convenient manner. Forexample, representations shown in AR device 200's display may vary basedon a relative distance to user 1201. For example, for individuals thatare close by, AR device 200 may display individual pictures, text,icons, etc. For individuals that are far away, AR device 200 may displaya nested icon that can be expanded upon selection, allowing selection ofa particular individual included in the nested icon. In addition, ARdevice 200 may be configured to filter information associated with otherindividuals such that only certain individuals and/or representationsare displayed at a given time. For example, AR device 200 may use afilter that shows only people that are within a certain distance rangefrom the user, or show only people within a particular azimuth range. Inanother embodiment, AR device 200 may show only certain user-selectedpeople (e.g., “favorites” specific to the user of AR device 200.).

In some embodiments, AR device 200 may be configured to be used tocommunicate with people within certain radiuses of the user. Forexample, AR device 200 may display a representation only for peoplewithin 100 m, 500 m, 1 km, etc. of the user. AR device 200 may also showrepresentations for people at farther distances, such as one mile,several miles, or even hundreds or thousands of miles, if desired. Forexample, user 1202 may look in a general direction and AR device 200 maydisplay a representation of a person in that direction or virtuallocation, even though the associated person is in a different city orcountry. User 1202 may nonetheless select the representation, and ARdevice 200 may establish a communication link between user 1202 and theperson in the different city or country.

Consistent with disclosed embodiments, a device, such as an AR device,may establish a communication link with another device based on anorientation of a user of the AR device, such as an orientation thatincludes the user looking at another individual. This feature may alignnaturally with the normal interaction of human communication, whereusers look at each other while talking. The ability to communicatethrough an AR device based on looking at another person may allow forimproved and efficient communication in a simple and natural manner,allowing people to more easily communicate in instances when it mayotherwise be difficult. For example, AR device 200 may allow forcommunication between users that can see each other but are too far awayto talk at a normal or acceptable speaking volume, users that wish tospeak privately or secretly even though others are present in a room,users that are relatively nearby but cannot see each other, and thelike).

Unlocking a Wearable Device

In some embodiments, a wearable augmented reality device may beconfigured to secure digital data associated with the device withpassword protection or similar security features. Wearable AR devices,in some cases, may not include typical input devices familiar to usersof personal computing devices, such as a mouse and a keyboard.Accordingly, wearable AR devices consistent with disclosed embodimentsmay be configured to allow a user to enter user credentials andpasswords using other inputs and methods that, in some embodiments mayallow for hands-free interaction. For example, in some cases including ahead mounted wearable AR device, user credentials and passwords may beentered based on the detected orientation of the user's head and/or thedetected gaze direction of the user's eyes, among other techniques.

This system of credential entry may avoid challenges and security riskstypically associated with attempting to enter passwords on a userdevice. Entering long strings of letters, numbers, and/or charactersrequires memorization of these strings, which may be difficult. Entry ofpasswords on touch screen devices leaves smudges or other visual indiciathat can be used to steal those passwords. And entering letters,numbers, and characters without external input devices may be difficultand time-consuming.

For example, consistent with some embodiments, a wearable device mayinclude a display and at least one sensor configured to provide anoutput indicative of a viewing direction of a user (e.g., an orientationof the head of a user and/or a gaze direction of the eyes of a user).The wearable device may further include at least one processing deviceconfigured to monitor a viewing direction of the user based on theoutput of the at least one sensor. On the display, the user may bepresented with a visual depiction of a series of points. In operationalscenarios where entry of passwords or other credentials is required, theuser may “dwell” or orient a depiction of a cursor (or reticle) overspecific points for specified periods of time. The “password” may thusrepresented by a predetermined pattern of the displayed points. To enterthe password, the user may follow the predetermined pattern, dwelling onthe first point, then the next, etc. until the pattern is completed.Based on the monitored viewing direction, the at least one processingdevice may be further configured to determine whether the user hassuccessfully “entered” the password.

As discussed above, in some embodiments, alternative in addition tomonitoring an orientation of the user's head, the processing device ofAR device 200 may be configured to monitor a user's gaze direction basedon output received from one or more eye tracking sensors. For example,the processing device may be configured to determine whether the userhas successfully “entered” the password based on the direction of theuser's gaze.

FIG. 15 illustrates an example of a user wearing an AR device consistentwith disclosed embodiments. As shown in FIG. 15, user 1501 is in anenvironment 1500 and is wearing an AR device 200, as discussed aboveregarding FIG. 2. AR device 200 may include a frame configured to securethe device to the head of user 1501. As discussed above, AR device 200may include a sensor array 125. Sensor array 125 may include one or moresensors configured to detect a viewing direction of user 1501. The oneor more sensors may include any one or more of a motion detector (e.g.,a gyroscope and/or an accelerometer), a camera, a location determinationdevice (e.g., a GPS device and/or an NFC reader), a magnetometer, an eyetracking sensor, and/or an orientation sensor (e.g., a theodolite,infra-red sensor). Such magnetometers (compasses) may detect orientationor a direction that a user is looking, accelerometers (e.g., three-axisaccelerometers) to detect the direction and speed of movementsassociated with AR device 200, or the like. For example, the viewingdirection of user 1501 may be defined according to a coordinate system,such as a three-dimensional coordinate system (x,y,z) having the originat a point on AR device 200, such as a central point of display 204, theposition at which IMU 201 is located, or any other reference point on ARdevice 200.

In some embodiments, positional processing module 378 shown in FIG. 3may be configured to store instructions that when executed by aprocessing device (e.g., microprocessor 208) of AR device 200, determinethe viewing direction of user 1501 based on output from the one or moresensors.

For example, in order to perform various tasks and/or access variousfeatures, such as locking and unlocking AR device 200, user 1501 maymove his or her head and/or eyes to various orientations within athree-dimensional coordinate system. As described above, one or moresensors included in AR device 200 may be configured to detect theviewing direction of user 1501. The sensors may then output informationindicative of the viewing direction to a processor device, such asmicroprocessor 208 shown in FIG. 2.

In some embodiments, the one or more sensors may be configured to detectthe viewing direction of user 1501 periodically. For example, the one ormore sensors may be configured to detect the viewing direction of user1501 at any suitable time interval (e.g. every second, multiple timesper second (2, 4, 6, 10 or more times per second), or once over thecourse of multiple seconds, etc.). Further, the one or more sensors maybe configured to output a detected angle of AR device 200 with respectto the horizontal plane (or other suitable reference plane) to aprocessing device (e.g., microprocessor 208). Alternatively, the one ormore sensors may be configured to output the detected coordinates of oneor more predetermined reference points of AR device 200 with referenceto a predetermined coordinate system. For example, the one or moresensors may be configured to detect coordinates of multiple points ofdisplay 204 in a predetermined three-dimensional coordinate systemhaving an origin set as the central point of AR device 200.

The processing device (e.g., microprocessor 208) of AR device 200 may beconfigured to monitor the viewing direction of user 1501 based on theoutput of the one or more sensors. For example, the processing devicemay be configured to determine the viewing direction of user 1501 basedon received coordinates of points of the device from the one or moresensors. In some embodiments, the processing device may be configured tomonitor both the viewing direction of user 1501 and the time duration(or “dwell time”) that the head and/or eyes of user 1501 stays in theviewing direction based on output received from the sensor. For example,if the time duration that the head and/or eyes of user 1501 stays in adetected viewing direction is less than a predetermined time threshold,the processing device may determine that user 1501 does not intend thedetected viewing direction to cause AR device 200 to take an action. Onthe other hand, if the time duration that the head of user 1501 stays ina detected viewing direction is greater than or equal to thepredetermined time threshold, the processing device may determine thatuser 1501 intends the detected viewing direction to cause AR device 200to take an action.

In the example illustrated in FIG. 15, user 1501's head (and byextension, AR device 200) is initially oriented at position 1502. Asdescribed above, position 1502 may be detected and processed in manyways within AR device 200, including as an angle relative to horizontalor vertical planes, or as a set of coordinates denoting the position ofAR device 200 within a three-dimensional coordinate system, as in theexample shown in FIG. 15. As AR device 200 progressively samples theorientation and position of the device, user 1501 changes theorientation of his or her head. At position 1503, user 1501 has tiltedhis or her head to assume a different orientation than position 1502.Later, a third position 1504 is registered. Although three positions aredepicted in the example shown in FIG. 15, a sequence for causing ARdevice 200 to perform a function may comprise any number of positions.Each of these orientations may be detected and registered by AR device200 and microprocessor 208 to execute commands or access variousfunctions and features of the device, as will now be described indetail. Alternatively, as discussed above, in other embodiments, theprocessing device of AR device 200 may be configured to monitor a user'sgaze direction based on output received from one or more eye trackingsensors. For example, the processing device may be configured todetermine whether the user has successfully “entered” the password basedon the direction of the user's gaze as the user's gaze moves in asequence of positions.

FIGS. 16A-16B illustrate an example of unlocking a lockable, wearable ARdevice consistent with disclosed embodiments. For example, renderingservices module 370, discussed above in reference to FIG. 3, may storesoftware instructions that, when executed by a processing device (e.g.,microprocessor 208), cause audio messages to be played, and/or causetext and/or graphics to be displayed to a user. As illustrated in FIG.16A, when AR device 200 is in a locked state, a virtual graphicalelement 1602 may be shown on display 204 of AR device 200 to indicatethis fact to the user. When in a locked state, such as one displayed inthe example of FIG. 16A, various features of AR device 200 may beinaccessible or otherwise unavailable in order to secure data on thedevice and/or data accessible via the device (e.g., data accessible vianetwork 130).

AR device 200 may be configured to enter an unlocked state uponreceiving a predetermined input from a user, such as a password. Thisprocess will be described in further detail below in association withFIGS. 17A-17K and 18. In some embodiments, AR device 200, via renderingservices module 370, may cause an array of graphical objects to be shownon display 204 of AR device 200. Rendering services module 370 mayfurther prompt the user to enter a predetermined input, such as apassword, by way of visual cues, audio cues, or both. Positionalprocessing module 378 may then be configured to detect “selection” ofthe password within the displayed graphical array by receiving outputindicative of the orientation of the head of the user from one or moresensors, such as sensor array 125, IMU 201, and/or GPS unit 202.

Upon receiving input of the password from the user AR device 200, viaprocessor device 123 and memory 124, may verify that the correctpassword has been entered. If the correct password has been received, ARdevice 200 may be configured to enter an “unlocked” state to provide theuser with access to information on display 204 of AR device 200.

In FIG. 16B, AR device 200 has verified correct password entry, and hasbeen unlocked. A virtual graphical element, such as virtual graphicalelement 1604 may be shown on display 204 of AR device 200 to indicate tothe user that the device has entered an unlocked state and that furtheroptions and resources may be accessible. For example, virtual menu 1606may become accessible to the user when AR device 200 is in an unlockedstate. Conversely, when AR device 200 is in a locked state, virtual menu1606 may be inaccessible to the user and may not appear on display 204of AR device 200. Alternatively, virtual menu 1606 may appear on display204, but may be rendered by rendering services module 370 in a mannerindicating that it is inaccessible. For example, virtual menu 1606 maybe blurred, faded, shrunk, etc.

Virtual menu 1606 may include one or more selectable elements eachassociated with one or more actions that may be taken by AR device 200upon selection by a user, such as user 1501, when AR device 200 is in anunlocked state. For example, virtual menu 160 may present options foruser 1501 to save AR content, share it with other users, rotate thedisplay of the AR content, zoom the display in or out, initiate a phonecall, change settings associated with a camera on AR device 200, or anyother options or tasks that user 1501 may select to provide input,request information, or initiate a task associated with AR device 200.

FIGS. 17A-17K illustrate an example of a series of interactive graphicaldisplays that may be rendered on display 204 of AR device 200 byrendering services module 370. The example graphical displays depictedin FIGS. 17A-17K illustrate an embodiment in which a user (such as user1501) may enter a password or other user credential in order totransition AR device 200 from a locked state to an unlocked state, asshown in FIGS. 16A-16B. The graphical displays depicted in FIGS. 17A-17Kare exemplary only and are not intended to be limiting. Any graphicalarrangement or display suitable for receiving input of information maybe used.

In some embodiments, AR device 200 may receive an indication that user1501 seeks to unlock the device. The indication may be a verbalindication from user 1501. In other embodiments, the indication mayinclude a tactile indication, or may be received based on positionalchanges detected by IMU 201, such as a shake or nod of the head of user1501. After receiving the indication, rendering services module 370 maypresent a graphical display to user 1501 on display 204. Any graphicaldisplay is contemplated that comprises some array of at least twographical objects. In some embodiments, the rendered graphical array maycomprise a plurality of points. The points may be arranged in anypattern and in a manner such that there is sufficient distance betweenany two points to allow them to be distinguishable.

For security purposes, the total number of points displayed may exceedthe number of inputs needed to enter the password or user credential.For instance, if the password to unlock AR device 200 consists of threecharacters, at least four points may be presented on display 204. Thisconfiguration may increase security, as it reduces the likelihood that asecond person could steal the password sequence simply by watching themovements of user 1501 as they enter the sequence. In alternativeembodiments, the total number of points displayed may exactly equal thenumber of inputs needed to enter the password or user credential, andadditional security may be provided by assigning various “dwell times,”or specific periods of time that the user must orient a cursor, reticle,or other selection device over a given point to select it as an input.In some embodiments, each point to be selected within a passwordsequence may be assigned a different dwell time to increase security.

In FIG. 17A, interactive graphical display 1700 a is shown. Graphicaldisplay 1700 a may represent an image shown to a user, such as user1501, on display 204 of AR device 200. Graphical display 1700 a may bedisplayed by rendering services module 370 upon receipt of an indicationby AR device 200 that user 1501 wishes to unlock the device. Graphicaldisplay 1700 a may comprise an initial arrangement of points forentering a password or other user credential. In the example presentedin FIG. 17A, a cursor 1702 is also displayed within graphical display1700 a. Cursor 1702 may be an icon, a trackable reticle, a pointer, orany other such cursor known in the art of computer graphics. Cursor 1702may be movable by user 1501 based on the orientation of the user's head,and thus the orientation of the device. Alternatively or in addition tomonitoring the orientation of the user's head, cursor 1702 may bemoveable by user 1501 based on the direction of the gaze of the user'seyes, Detection of user movement and translation of the movement intocursor motion on display 204 may be processed by rendering servicesmodule 370, visual processing module 374, and positional processingmodule 378. To enter the password to unlock AR device 200, user 1501 maymove or orient cursor 1702 over specific points within graphical display1700 a for specific periods of time.

In FIG. 17B, interactive graphical display 1700 b is shown. Graphicaldisplay 1700 b may also represent an image shown to a user, such as user1501, on display 204 of AR device 200. In the example illustrated inFIG. 17B, user 1501 has moved cursor 1702 over one of the pointsdisplayed in graphical display 1700 b. In some embodiments, AR device200, via visual processing module 374 and positional processing module378, may determine an “input value” associated with the point. Thedetermined input value may comprise, for example, the length of time(“dwell time”) that cursor 1702 remains oriented over the particularpoint. In other embodiments, the input value may be determined in otherways, such as the accuracy of the alignment of cursor 1702 over thepoint, the speed at which the cursor is moved to the point, etc.

The input value, such as the dwell time, may be represented graphicallyto user 1501 within graphical displays such as graphical display 1700 bvia rendering services module 370. A graphical display associated withthe point may be changed by rendering services module 370 to representthe increasing input value. For example, the graphical display coulddisplay a sequence of numbers from 1 to 10 as the cursor dwells on theparticular point for specific time periods. Alternatively, the displayedsequence of numbers could progress downwards from 10 to 1. In otherembodiments, colors could be used instead of numbers; for example, thegraphical display associated with the particular point could cyclethrough the colors of the rainbow from red to violet. In still furtherembodiments, the graphical display could cycle progressively throughletters of an alphabet, for example the English language Latin alphabet,a Cyrillic alphabet, or characters of Chinese, Japanese, or Koreanlanguages. These examples are not intended to be limiting, and anygraphical display may be used to indicate increasing input values for aparticular point.

FIGS. 17C-17F illustrate an example advancement of input value for aparticular point within a graphical display. In the example of FIGS.17C-17F, cursor 1702 dwells on the point to which it was moved in FIG.17B. For demonstration purposes, the input value required for thatparticular point in the entry of the password sequence is 4. In FIG.17C, user 1501 has oriented his or her head in a manner to permit cursor1702 within interactive graphical display 1700 c to dwell on theparticular point for a specified period of time, or dwell time. In theexample of FIG. 17C, increasing input values are illustrated bydisplaying increasing numbers. Once the predetermined dwell time for theparticular point contacted by cursor 1702 has passed, the input valuefor that point advances to 1. As cursor 1702 continues to dwell on thepoint, the input value continues to incrementally increase each time aspecified period of time has passed. In FIG. 17D, the input value hasincreased to 2 within interactive graphical display 1700 d. In FIG. 17E,the input value has increased to 3 within interactive graphical display1700 e. Finally, in FIG. 17F, the input value has reached 4 withininteractive graphical display 1700 f, which is the desired input valuefor the first of the input points in the password sequence pattern.Accordingly, user 1501 has successfully input the first element of thepattern in FIG. 17F.

In the examples of FIGS. 17A-17K, the password to be entered has threeinputs, so user 1501 must move cursor 1702 to the next input point inthe sequence. In FIG. 17G, interactive graphical display 1700 g isshown. Graphical display 1700 g may represent an image shown to a user,such as user 1501, on display 204 of AR device 200. In the exampleillustrated in FIG. 17G, user 1501 has moved cursor 1702 over a secondpoint in the array of points displayed in graphical display 1700 g. Thispoint may represent the location of the second point within thethree-point object selection sequence pattern. FIG. 17H illustrates theinput value increasing within interactive graphical display 1700 h forthe second point as user 1501 continues to dwell cursor 1702 on thepoint.

In FIG. 17I, interactive graphical display 1700 i is shown. In theexample illustrated in FIG. 17I, user 1501 has reached the requiredinput value/dwell time for the second point in the password sequencepattern, which in this example is 8. User 1501 then is moving cursor1702 towards the location of the third and final point in the sequence.In FIG. 17J, user 1501 has moved cursor 1702 over this third point inthe array of points displayed in graphical display 1700 j. FIG. 17Jfurther illustrates the input value again increasing within interactivegraphical display 1700 j for this third point as user 1501 continues todwell cursor 1702 on it.

Each input value is specific for its assigned point. For example, in theillustration of FIG. 17J, the first point in the password sequencepattern has an assigned input value of 4, and the second point in thesequence has an assigned input value of 8. The input value must matchits assigned point. For example, placing an input value of 8 on thefirst point would constitute an incorrect or invalid password entry, aswould placing an input value of 4 on the second point.

In some embodiments, after the correct number of points have beentriggered by user 1501 for the appropriate amounts of time, renderingservices module 370 may generate a button or new point that allows user1501 to enter or submit the password for verification. Alternatively,the password may be automatically entered after successful input of thecorrect input values at the correct input points. Either successfulsequence may lead to FIG. 17K, which indicates via interactive graphicaldisplay 1700 k that the password has been successfully entered, and thatwearable AR device 200 is now in an unlocked state. Although text isshown in FIG. 17K, AR device 200 may display any suitable message and/orgraphic representing that AR device 200 has been unlocked. After ARdevice 200 has been unlocked, rendering services module 370 may thenconfigure display 204 to permit user 1501 access to various features ofAR device 200, such as virtual menu 1606 in the example of FIG. 16B.

FIG. 18 is a flowchart of an exemplary process 1800 for providing a userwith access to secured information on the display of a lockable,wearable AR device consistent with disclosed embodiments. As an example,one or more steps of process 1800 may be performed by AR device 200shown in FIG. 2. For example, in process 1800 a sensor of AR device 200(such as positional processing module 378) may detect an orientation ofthe head of user 1501 and provide a corresponding output to a processingdevice of AR device 200. User 1501 may move his or her head in order tocorrespondingly move a cursor or reticle within the display for purposesof entering a password or other user credential to unlock AR device 200.The processing device of AR device 200 may track the pattern of movementof the user, and unlock the device if the pattern matches apredetermined pattern used as the password.

As another example, in process 1800, a sensor of AR device 200 maydetect a gaze direction of the eyes of user 1501 and provide acorresponding output to a processing device of AR device 200. User 1501may move his or her eyes in order to correspondingly move a cursor orreticle within the display for purposes of entering a password or otheruser credential to unlock AR device 200. The processing device of ARdevice 200 may track the pattern of movement of the user, and unlock thedevice if the pattern matches a predetermined pattern used as thepassword.

At step 1810, AR device 200 may monitor a viewing direction of user 1501(e.g., an orientation of the head of user 1501 and/or a gaze directionof the eyes of user 1501) based on output of at least one sensorassociated with the AR device. For example, the processing device (e.g.,microprocessor 208) of AR device 200 may execute instructions ofpositional processing module 378, discussed above in relation to FIG. 3,to perform this step. The sensor may be included in AR device 200 andmay be configured to provide an output indicative of the viewingdirection of the user. For example, the sensor may be configured toprovide an output indicative of the viewing direction of the user bytracking a pattern of movement of an orientation of the head of theuser. As another example, the sensor may be configured to provide anoutput indicative of the viewing direction of the user by tracking agaze of the user's eyes.

User 1501 may be presented with a series of interactive graphicaldisplays, such as displays 1700 a-1700 k described above in associationwith FIGS. 17A-17K. The graphical displays may be presented on display204 of AR device 200, and may comprise an array of graphical objects,such as points, icons, etc. A subset of these graphical objects may beselected by user 1501 as part of a password, and AR device 200 maydetect that selection by tracking the pattern of movement of user 1501'shead.

In some embodiments, the pattern of movement may include two or moremovements along different directions as user 1501 maneuvers thegraphical display array to select the particular objects associated withthe password. In some embodiments, the two or more movements alongdifferent directions may comprise a leftward movement and a rightwardmovement. In other embodiments, the two or more movements alongdifferent directions may include an upward movement and a downwardmovement.

The “tracking” of user 1501's movement in step 1810 may include trackingthe position of a cursor or reticle on the display 204, which is movableby user 1501 in concert with the orientation of his or her head and/oreyes. This cursor may be similar to the cursor 1702 depicted in FIGS.17A-17K.

At step 1820, AR device 200 may determine that the tracked pattern ofmovement for user 1501's head and/or eyes matches a predeterminedpattern of movement—namely, the preset password configured for purposesof unlocking a lockable AR device. As discussed above in associationwith FIGS. 17A-17K, entry of the password may comprise having user 1501move a cursor or reticle, such as cursor 1702, around the interactivegraphical display to select the two or more objects or points within thegraphical array. A processing device associated with AR device 200, suchas microprocessor 208, may further determine a specific input value ateach particular point in the sequence based on how long user 1501maintains the orientation of his or her head and/or eyes to dwell onthat point. Each point in the password sequence may be configured torequire a specific input value in order to constitute a successfulentry. Accordingly, AR device 200 may detect an interception of theposition of the cursor 1702 with an area of the interactive graphicaldisplay (e.g., displays 1700 a-1700 k) associated with at least twoobjects displayed within an array of objects. AR device 200, viamicroprocessor 208, may further monitor dwell time of user 1501 on eachof the given objects. Detection of a successful selection of aparticular point within a password sequence pattern may occur when thedwell time exceeds a predetermined amount of time configured for each ofthe points.

The detection process associated with step 1820 may be modified toprovide additional security features. In some embodiments, in order tominimize the opportunities for an outside entity to decipher thepassword by watching the head movement and/or eyes of the user theposition of the plurality of points can be altered. For example, thepositioning of the entire array may be shifted within the interactivegraphical display each time the points are displayed. In otherembodiments, the position of individual points within the array may beshifted. This may create some additional ambiguity in the motion of ARdevice 200.

In other embodiments, AR device 200 may ignore the required order ofentry of the individual points within the password sequence pattern,permitting user 1501 to enter the individual inputs in any order, and ina different order each time if desired. In these embodiments, theability of an outside observer to detect repeated, similar headmovements and/or eye movements may be substantially reduced.Alternatively, the required order may cycle in a random order each timethe device is unlocked. As an additional feature, password inputs mayoverlay on one another at a single point; in other words, user 1501could enter input at a particular point within the array, then return tothat same point later in the sequence entry to enter a second input.

Additional security measures may be included. In some embodiments,specific zones or travel paths may be configured for correct passwordentry. For example, user 1501 may be required to pass cursor 1702through certain points or between certain regions of the screen whilemoving between the input-required points. In these embodiments, if ARdevice 200 does not detect that cursor 1702 passes through theadditionally configured positions, then the inputs, even if correct,would not constitute an acceptable password. These configured “paths”may or may not be visible to the user on display 204 while entry is inprogress.

In still other embodiments, rendering services module 370 may configureicons to be used as the objects within the array instead of discrete,identical points. Input value could then be indicated as described aboveby cycling numbers on top of each icon as a function of dwell length.The icons may be shuffled around each time the device is unlocked,increasing security. For example, the display may be configured todisplay icons representing animals as the points comprising thepassword. The password may then comprise, e.g., “chicken for 5 (dwelltime),” “dog for 3,” and “horse for 6.” This password could therefore beentered no matter where those particular icons were displayed within thegraphical array of objects. Alternatively, the display may be configuredin a manner where the icons move on the screen, and password entry maybe configured such that user 1501 is to dwell on a particular locationuntil the desired, correct icon reaches that position (or any otherparticular position).

In some embodiments, AR device 200 may be equipped with additional inputdevices, such as a pointer, stylus, mouse device, etc. The additionalinput devices may provide input to AR device 200 via, for example,communications interface 121. In these embodiments, user 1501 may use acombination of dwell time and the input device(s) to select an icon orpoint. In some embodiments, the selection of the points or icons may beaccomplished via movement of the head of user 1501 and/or via movementof the eyes of user 1501, and confirmation of the entry may be effectedby use of a second input device (e.g., a fingerprint recognitiondevice). For example, AR device may be equipped with a fingerprintrecognition device that can recognize one or more fingerprints (e.g., athumbprint of a user). In some embodiments, the fingerprint recognitiondevice may be used to facilitate the unlocking of AR device 200 withoutthe need of monitoring the orientation of the head of the user. In otherembodiments, the fingerprint recognition device may be used inconnection with entry of a password via tracking the orientation of thehead of the user of AR device 200, as described above. Thus, thefingerprint recognition device may serve as a second factor in a twofactor authentication process. In addition, in some embodiments, as analternative (or in addition) to monitoring the orientation of the headof the user of AR device 200, AR device 200 may include an eye trackerdevice, which may monitor an eye of the user of AR device 200 in orderto detect a password entered when the user looks at certain locations.

At step 1830, AR device 200 may transition from a locked state to anunlocked state to provide user 1501 with access to information ondisplay 204 of the device. This process is illustrated above inassociation with FIGS. 16A-16B. AR device 200 may verify that theentered password pattern of movement is correct via comparison with astored value within memory 124. Alternatively, AR device 200 may comparethe entered password to a stored value on a remote server via network130, such as memory 112 or database 113 of server system 110.

After transitioning to an unlocked state, user 1501 may gain access toany or all subset of features, functions, and programs stored within ARdevice 200. As an example, after being unlocked, AR device 200 maydisplay a virtual menu such as virtual menu 1606 of FIG. 16B. In someembodiments, AR device 200 may be configured such that multiple usersmay be capable of using the device, each with a different password. Inthese embodiments, the display 204 may be configured to providedifferent options and functions to each individual user (e.g., optionsand functions associated with a user profile). Various permissions toaccess different options and functions may be configured and managedremotely, such as via server system 110.

Providing Task-Based Instructions

In one embodiment, the progress of a task assigned to a particular userof a wearable device may be monitored. As steps in the task arecompleted, information relating to the next steps may also be passed tothe user. The information relating to the task and next steps in thetask may be received by the wearable device over or a network and/orretrieved from a memory of the wearable device.

For example, consistent with some embodiments, a wearable deviceprovides a user of the device with task-based instructions. The devicecomprises a display, a network interface, and a data input deviceconfigured to capture information relating to progress of a task. Thedevice further comprises at least one processing device configured toprovide to a server system, via the network interface, the capturedinformation relating to progress of the task; receive from the serversystem, via the network interface, information relating to a next stepin the task; and cause the information relating to the next step in thetask to be shown on the display.

Referring to FIG. 1, user system 120 (e.g., AR device 200) maycommunicate with server system 110 via network 130 to assist a user tocomplete a task. For example, AR device 200 may send a request to serversystem 110. The request may identify information, such as instructions,needed for performing a task via AR device 200. AR device 200 may alsosend captured information relating to the task, such as an indication ofprogress of the task, an identifier of AR device 200, a location of ARdevice 200, etc., to server system 110 with the request. Server system110 may store information relating to the task, such as instructions fora user of AR device 200 to perform the task, in database 113 and/ormemory 112. In some embodiments, server system 110 may retrieve thetask-related information from a remote server accessed over network 130and store the retrieved information in database 113 and/or memory 112.Server system 110 may gather the requested information related to thetask and send it to AR device 200 via network 130. In some embodiments,server system 110 may include legacy IT resources and/or constitute anenterprise resource planning (ERP) system (e.g., such as an ERP systemavailable from SAP and Oracle).

In some embodiments, server system 110 may provide data to AR device 200via network 130 according to a store-and-forward data model. In such animplementation, AR device 200 may not need to maintain connectivity withnetwork 130. For example, AR device 200 may download tasks and workflowsprior to the commencement of the work. AR device 200 may then provideinstructions (e.g., images, video, text, audio, etc.) for one or moresteps of the tasks and store the result or results of the task while ARdevice 200 is offline and not available and/or not in communication withnetwork 130. The stored data may be stored in an encrypted format.Further, after AR device 200 resumes connectivity with network 140, ARdevice 200 may send the result or results of the task to server system110.

FIG. 19 illustrates an example diagram 1900 of a workflow for providinga user of an AR device with task-based instructions consistent withdisclosed embodiments. For example, AR device 200 may receive user inputto initiate a task that requires a series of steps to be taken by thedevice. The user input can be a command that causes AR device 200 totake a certain action, accomplish a certain task, provide certaininformation to the user, or any other types of actions associated withAR device 200. A data input device (e.g., camera 203, microphone 205) ofAR device 200 may be configured to capture information relating to theprogress of the task in response to the received user input. Forexample, the data input device may be an image capture device configuredto acquire images, a voice recognition module configured to receiveinformation from voice command of the user, a physical button for theuser to press and confirm certain operations, a touch sensitive surfacethat can detect user gesture inputs selecting information, or any otherinput device that can be configured to capture information relating tothe task.

As shown in FIG. 19, AR device 200 may send the captured informationrelating to progress of a task to server system 110 at step 1901. Forexample, at the initial stage of performing the task, AR device 200 maysend a request for initiating the task to server system 110, indicatingthat the task is initiated by the user and has not been performed. Insome embodiments, at the initial stage of performing the task, AR device200 may send an identification code of the specific task that isrequested, an identifier (e.g., any alphanumeric information) of ARdevice 200, and/or a current location (e.g., GPS coordinates, a streetaddress, etc.) of AR device 200, to server system 110 such that serversystem 110 may identify the related task that is requested and alsoretrieve appropriate task-related information based on propertiesassociated with AR device 200. For example, server system 110 maydetermine whether AR device 200 or a user of AR device 200 is authorizedto perform the task based on the received identifier of AR device 200.If AR device 200 is not authorized to perform the task, server system110 may send a message to AR device 200 informing the device thatinformation related to the task will not be provided and instructing thedevice to stop the task. In other embodiments, AR device 200 may beauthenticated via other information, such as a username and passwordcombination submitted by a user.

In some embodiments, server system 110 may identify the requiredhardware and/or software configurations of an AR device in order toperform the task and determine whether the AR device has the capabilityto perform the task. For example, server system 110 may store thehardware and/or software configurations of AR device 200 in database 113and may retrieve the configurations of AR device 200 based on thereceived identifier of AR device 200. As another example, AR device 200may send its hardware and/or software configurations to server system110 during the initial stage of performing the task such that serversystem 110 may determine whether AR device 200 is capable of performingthe task. If server system 110 determines that AR device 200 is notcapable of performing the task, server system 110 may send a message toAR device 200 indicating that the device is not capable of performingthe task and instructing the device to stop the task. On the other hand,if server system 110 determines that AR device 200 is capable ofperforming the task, server system 110 may proceed and provide thetask-related information to AR device 200.

In some embodiments, server system 110 may identify that additionalinformation related to AR device 200 is required in order to provide thetask-related information to the device. For example, server system 110may identify that information related to hardware and/or softwareconfigurations of AR device 200 is required in order to determinewhether the requested task can be performed by the device. In thissituation, server system 110 may send a message to AR device 200requesting the additional required information via network 130. Whenadditional information is received from AR device 200, server system 110may store the received information in database 113 such that serversystem 110 may retrieve the information directly for future tasks. Insome embodiments, server system 110 may require the user of AR device200 to input a passcode in order to authenticate the identity of theuser. For example, server system 110 may send a message to AR device 200requesting an input of the passcode, and if an incorrect passcode isreceived from AR device 200, server system 110 may send a message to ARdevice 200 informing the device that the passcode is incorrect andrequires AR device 200 to resend the passcode. After a number ofunsuccessful attempts, server system 110 may instruct AR device 200 tostop the task. If the correct passcode is received from AR device 200,server system 110 may proceed to provide the task-related information toAR device 200.

At step 1902, server system 110 may provide information relating to anext step in the task to AR device 200. After receiving the informationrelating to the next step in the task, the processing device of ARdevice 200 may cause the information to be shown on display 204. Asillustrated in FIG. 19, an example of the next step in the task is totake an image of the user's current field of view. Server system 110 maysend the information relating to the next step in the task to AR device200, and the processing device of AR device 200 may cause task-assistinformation 1905 (e.g., the message “Take an Image”) to be shown ondisplay 204. While a text-based instruction is used in this example, theinformation relating to the next step in the task may be provided to ARdevice 200 in a variety of forms, such as a text-based instruction, agraphical-based instruction, an audible instruction, a videoinstruction, or a combination thereof.

Task-assist information 1905 may be shown at a predetermined position ondisplay 204, such as the center of display 204, an upper portion ofdisplay 204, or any other suitable location. The predetermined locationfor displaying task-assist information 1905 may be pre-configured byuser 401 through a user interface of AR device 200. For example, display204 may allow user 401 to select a location of display 204 at whichtask-assist instruction 1905 will be displayed. In some embodiments,after task-assist information 1905 is displayed at a pre-set location,user 401 may be allowed to drag task-assist information 1905 to anotherlocation on display 204 that is preferred by user 401. For example, theprocessing device of AR device 200 may set the new location as a defaultlocation for displaying task-assist information 1905, and displaytask-assist information 1905 at the new location when AR device 200receives information relating to a next step in a task. In otherembodiments, the location of task-assist information 1905 may be lockedto a predetermined position of display 204 such that user 401 cannotreposition the task-assist information.

In response to the displayed information relating the next step in thetask, the user of AR device 200 may perform actions to continueexecution of the task based on the displayed information. For example,the user may take an image of the current field of view according totask-assist information 1905. In some embodiments, the processing deviceof AR device 200 may cause an action interface, such as a command oricon, to be displayed on display 204 subsequent to or concurrently withthe display of task-assist information 1905. The command or icon may beassociated with actions of the next step in the task indicated in thetask-assist information in order to facilitate the user to take the nextstep. For example, a camera icon 1906 may be displayed on display 204,and the user of AR device 200 may perform the next step in the task(e.g., taking a picture) by simply selecting the icon. In someembodiments, the information relating to the next step in the task mayinclude more than one option that the user may take to continue thetask, and more than one command or icon may be displayed on display 204.For example, the information relating to the next step in the task mayindicate that the user may either take an image of the current field ofview or input the current location of the user. Accordingly, an inputfield may be displayed along with a camera icon on display 204, and theuser may either select the camera icon or input the user's currentlocation in the displayed input field to proceed. In other embodiments,the processing device of AR device 200 may not cause an action interfaceto be shown on the display and may require the user to initiate the nextstep in the task by opening a menu or pressing a button of the AR deviceto perform the action in the next step.

In some embodiments, the user of AR device 200 may confirm completion ofa step of the task by taking certain actions associated with a datainput device of AR device 200. For example, the data input device (e.g.,microphone 205) may include a voice recognition module configured todetermine whether the user has audibly confirmed completion of a taskstep. In such an embodiment, the user may issue a voice command toconfirm completion of the task step when the step is completed. Asanother example, the user may press a button on AR device 200 to confirmcompletion of a task step. In another example, a text command, icon, orcheck-box may be displayed on display 204 for the user to select andconfirm completion of a task step. In other embodiments, the processingdevice may determine completion of a task step by detecting that thestep provided by server system 110 has been performed, and thus wouldnot require confirmation by the user of the completion of a task step.

After the next step in the task is performed, the data input device ofAR device 200 may be configured to capture information relating to theprogress of the task. For example, the data input device may cause theprogress of the task to be updated when a certain step is completed.Referring to FIG. 19, after the user of AR device 200 takes the imageaccording to task-assist information 1905, the data input device (e.g.,the camera of the device) may capture information indicating that theimage has been taken and provide this information to the processingdevice of AR device 200. The processing device may correspondinglyupdate the progress of the task to reflect that the next step iscompleted. The data input device may also provide the captured data tothe processing device. For example, camera 203 of AR device 200 may savethe captured image data in a digital file that is accessible by theprocessing device.

At step 1903, the processing device of AR device 200 may send theupdated information relating to the progress of the task, such as thecompletion of the next step provided in step 1902, to server system 110via network 130. The processing device may also send the captured datain performing the step in the task, e.g., the captured image file, toserver system 110 if server system requires the captured data to proceedin the task. For example, after receiving the updated informationrelating to the progress of the task, server system 110 may send amessage to AR device 200 requesting for the captured data in performingthe step in the task, and AR device 200 may send the captured data toserver system 110 via network 130.

After receiving the updated information relating to the progress of thetask, server system 110 may determine whether the task is completed oradditional steps of the task remain incomplete. In some embodiments,server system 110 may determine the required steps by accessing database113, compare the required steps and the completed steps, and determinewhether the task is completed. In other embodiments, server system 110may forward the information related to the progress of the task to aremote server that may determine whether the task is completed. Forexample, the remote server may provide server system 110 with additionalsteps to be taken by AR device 200, or send a confirmation message toserver system 110 indicating that the task is completed.

If additional steps are required, server system 110 may provideinformation relating to the next step in the task to AR device 200,similar to step 1902. AR device 200 may display information relating tothe next step in the task. After the next step is completed, AR device200 may send updated information relating to the progress of the task toserver system 110, similar to step 1903. In other words, steps 1902 and1903 may be repeated until all the steps of the task are completed. Insome embodiments, server system 110 may provide information includingmultiple steps (e.g., a list of the required steps) to perform a task inone or more messages transmitted to AR device 200. AR device 200 maycorrespondingly display the required steps on display 204. After thesteps in the task are completed, AR device 200 may send updatedinformation relating to the progress of the task to server system 110,indicating that the steps have been completed.

At step 1904, when the steps of the task are completed, server system110 may send an indication to AR device 200 indicating that the task iscompleted. In some embodiments, server system 110 may provide automaticconfirmation of step and/or task completion (e.g., based on images,text, and/or video transmitted to server system 110). For example,server system 110 may analyze information received from AR device 200after a step has been completed and determine whether the step wascompleted correctly. If so, server system 110 may automatically confirmcompletion and my send the next task instruction to AR device 200 basedon this determination. If the step was not completed correctly, serversystem 110 may provide an instruction to clarify and/or correct theprior step and transmit that information to AR device 200.

In addition, in some embodiments, an operator of a workstation (e.g.,computer, smartphone, tablet, etc.) may receive data over network 130from server system 110 regarding a completed step, and the operator maydetermine whether the step was completed correctly. For example, theoperator may review text, images, and/or video transmitted from ARdevice 200 regarding a step performed by the user of AR device 200. Ifthe step was completed correctly, the operator may confirm the correctcompletion, and the operator may then select and send the nextinstruction AR device 200. If the step was not completed correctly, theoperator may send supplemental information to AR device 200 to clarifyand/or correct the step.

After receiving an indication indicating that a task has been completed,AR device 200 may display a message 1907 on display 204 informing theuser that the task has been completed. In some embodiments, a progressbar may be shown on display 204 informing the user of the progress ofthe task. The percentage of the task being completed may be increased inthe progress bar each time a step of the task is completed. After thetask has been completed, the processing device of AR device 200 maycause the progress bar to show that 100 percent of the task iscompleted.

In some embodiments, an operator may modify and/or select theinformation that is transmitted to AR device 200 relating to a step in atask. For example, a workstation (e.g., computer, smartphone, tablet,etc.) may receive data over network 130 from server system 110. Anoperator of the workstation may monitor information (e.g., text, images,and/or video) received by server system 110 from AR device 200. Theoperator may then determine whether a task is complete and/or send thenext instruction. For example, via a user interface of the workstation,the operator may send one or more auxiliary instructions to supplementand/or modify instructions being sent to AR device 200. The operator maysend these auxiliary instructions to, for example, correct errors thatmay have been made and/or to address unexpected events.

In addition, in some embodiments, the operator may annotate informationprovided to display 204 of AR device 200. The operator may access aworkstation (e.g., computer, smartphone, tablet, etc.) and use a userinterface to draw annotations that are transmitted to display 204 of ARdevice 200. For example, the operator may highlight areas of interestassociated with a workplace, the user's environment, and/or an object inthe user's environment. As another example, the operator may circle apart to be worked on (e.g., point to a fastener), or identify problemsby circling or pointing to objects in the user's field of view. Serversystem 110 may receive these annotations from the workstation overnetwork 130 and then transit the annotations with or supplemental toinformation being transmitted to AR device 200. Accordingly, AR device200 may display the annotations received from the operator of theworkstation when displaying any of the instructions to complete a taskon display 204 of AR device 200.

FIG. 20 illustrates another example diagram 2000 of a workflow forproviding a user of an AR device with task-based instructions consistentwith disclosed embodiments. For example, AR device 200 may receive userinput to initiate a task that includes a series of steps. A data inputdevice (e.g., camera 203, microphone 205) of AR device 200 may beconfigured to capture information relating to the progress of the taskin response to the received user input.

At step 2001, AR device 200 may send the captured information relatingto progress of a task (e.g., a request for providing task-basedinstructions relating to the task) to server system 110. For example, ARdevice 200 may send a request to initiate the task to server system 110.AR device 200 may also send an identifier of the specific task that isrequested, an identifier of the AR device, and/or a current location ofthe AR device, to server system 110.

At step 2002, server system 110 may provide a video instruction forperforming a next step in the task to AR device 200. Alternatively, ARdevice 200 may retrieve a video instruction for performing a next stepin the task from a memory included in AR device 200. The videoinstruction may include a video presentation of how the next step in thetask should be conducted. After receiving the video instruction forperforming the next step in the task, the processing device of AR device200 may cause a video icon 2004 to be shown on display 204, indicatingto the user of AR device 200 that a video instruction for performing thenext step in the task is available. Video icon 2004 may be shown at apredetermined position on display 204, such as the center of display204, an upper portion of display 204, or any other suitable location. Insome embodiments, a text notification may be displayed in place of or inaddition to the video icon to inform the user that a video instructionis available. In some embodiments, an audio notification, such as a ringtone, may alert the user that a video instruction is available. Itshould be understood that while a video instruction is used in thisexample, the information relating to the next step in the task may bepresented in other forms, such as an audio instruction, a textinstruction, a graphic instruction, or a combination thereof, withoutdeparting from the spirit of the present disclosure.

The user of AR device 200 may select video icon 2004 to play the videoinstruction. In some embodiments, the video instruction may includemultiple steps in the task or all of the steps in the task. When thenext step in the task is completed, the processing device of AR device200 may cause a text message 2005 shown in the display, indicatingcompletion of the task step. In some embodiments, text message 2005 maybe selectable, and the user of AR device 200 may select text message2005 to confirm that the task step is completed (e.g., by looking in thedirection of text message 2005 for a certain period of time, touchingthe portion of the display that text message 2005 is shown, moving areticle to intercept with text message 2005, pressing a button on ARdevice 200, or the like). If the video instruction includes multiplesteps in the task, the selectable text message may be displayed afterall the steps provided in the instruction are completed, and the user ofAR device 200 may select text message 2005 to confirm that all the stepsprovided in the instruction are completed by performing the operationsdescribed above.

At step 2003, AR device 200 may send an indication to server system 110that the provided task step is completed. AR device 200 may send theindication when AR device 200 is in communication with server 110. Afterreceiving the indication, server system 110 may determine whether thereare additional steps in the task to be performed by AR device 200.Alternatively, AR device 200 may retrieve a video instruction forperforming a next step in the task from a memory included in AR device200. If there are additional steps in the task, steps 2002 and 2003 maybe repeated until all the steps in the task have been performed. Ifserver system 110 determines that all the steps in the task have beenperformed, server system 110 may provide an indication to AR device 200indicating that the task is completed. Alternatively, AR device 200 maydetermine whether the task is completed based on data stored in a memoryof AR device 200.

FIG. 21 is a flowchart of an exemplary process 2100 for providingtask-based instructions via an AR device consistent with disclosedembodiments. As an example, one or more steps of process 2100 may beperformed by AR device 200 shown in FIG. 2.

At step 2110, AR device 200 may capture information relating to progressof a task. For example, a data input device (e.g., camera 203,microphone 205, display 204) of AR device 200 may capture informationrelating to progress of a task and provide the information to aprocessing device of AR device 200. For example, the data input devicemay include an image capture device configured to acquire image dataduring performance of steps in the task.

At step 2120, AR device 200 may provide to a server system, via anetwork interface, the captured information relating to progress of thetask. For example, the processing device of AR device 200 may executeinstructions of network interaction services 372 to perform this step.The network interface may include one or more of an Internet connection,a cellular connection, a WAN connection, or a LAN connection, or anyother type of network connection. In some embodiments, AR device 200 mayalso provide an identifier of AR device 200 to the server system. Insome embodiments, the server system may include an enterprise resourceplanning system.

At step 2130, AR device 200 may receive from the server system, via thenetwork interface, information relating to a next step in the task. Forexample, the processing device of AR device 200 may execute instructionsof network interaction services 372 to perform this step. Alternatively,AR device 200 may retrieve a video instruction for performing a nextstep in the task from a memory included in AR device 200. Theinformation received from the server may include a video instruction, atext-based instruction, a graphical-based instruction, or an audibleinstruction for performing the next step in the task. At step 2140, ARdevice 200 may cause the information relating to the next step in thetask to be shown on the display. For example, the processing device ofAR device 200 may execute instructions of rendering services module 370to perform this step. The information relating to the next step in thetask may be shown at a predetermined location on the display. A user ofAR device 200 may subsequently follow the information and cause the nextstep in the task to be performed. In some embodiments, the data inputdevice may include a voice recognition module configured to determinethat the user has audibly confirmed completion of a task step. In someembodiments, the data input device may include a button for the user topress to confirm completion of a task step. In some embodiments, theprocessing device of AR device 200 may cause an indication to be sent tothe server system after completion of the task step.

Controlling an On-Board Component

In one embodiment, an interface is disclosed for controlling an on-boardcomponent without relying upon voice or button activated controls.

For example, consistent with an embodiment, a wearable device maycontrol one or more operations of an on-board component. The device maycomprise a display and at least one sensor configured to provide anoutput indicative of a viewing direction of a user (e.g., an orientationof a head of a user and/or a gaze direction of the eyes of a user). Thedevice may further comprise at least one processing device configured tocause at least one graphical icon associated with a control of at leastone aspect of the on-board component to be shown on the display suchthat the user perceives the location of the at least one graphical iconas fixed relative to real world coordinates; determine, based on theoutput of the at least one sensor, whether the user is looking in adirection of the at least one graphical icon; and initiate the controlof the at least one aspect of the on-board component when the user isdetermined to be looking in the direction of the at least one graphicalicon.

FIG. 22 illustrates an example of a user wearing an AR device configuredto control an operation of an on-board component consistent withdisclosed embodiments. As shown in FIG. 22, user 2201 is in anenvironment 2200 and is wearing AR device 200, as discussed aboveregarding FIG. 2. The processing device of AR device 200 may beconfigured to display a graphical icon and/or text associated withcontrolling one or more on-board components on display 204. For example,the processing device may be configured to display a graphical iconassociated with controlling an on-board component on display 204 inresponse to detecting that a button of AR device 200 is pressed by user2201. In some embodiments, user 2201 may cause AR device 200 to operatean on-board component by changing the orientation of his or her headand/or eyes. For example, user 2201 may look in a direction of thegraphical icon associated with controlling the on-board component for acertain period of time, and cause AR device 200 to initiate the controlof an aspect of the on-board component.

As discussed above, AR device 200 may include sensor array 125, whichmay include one or more sensors configured to detect a viewing directionof user 2201. For example, the processing device of AR device 200 may beconfigured to determine whether user 2201 is looking in a direction ofthe graphical icon based on the monitored orientation of the head and/oreyes of user 2201. In some embodiments, positional processing module 378shown in FIG. 3 may be configured to store instructions that, whenexecuted by a processing device (e.g., microprocessor 208) of AR device200, determine whether user 2201 is looking in a direction of thegraphical icon on output from the one or more sensors.

For example, the processing device may determine the direction of theuser's sight line based on the monitored orientation of the head of user2201 and/or the gaze direction of the eyes of user 2201. The processingdevice may identify a region on display 204 that includes the graphicalicon associated with controlling the on-board component. The identifiedregion may be the same as or slightly larger than the area the graphicalicon occupies. If the direction of the user's sight line falls into theidentified region, the processing device may determine that user 2201 islooking in the direction of the graphical icon and initiate operation ofthe on-board component. On the other hand, if the direction of theuser's sight falls outside the identified region, the processing devicemay determine that user 2201 is not looking in the direction of thegraphical icon associated with control of the on-board component, andthus, may not initiate control of the on-board component.

In some embodiments, the processing device may be configured todetermine a time duration that user 2201 looks in a direction of thelocation of the graphical icon associated with controlling the on-boardcomponent based on output received from the sensor. For example, thesensors of AR device 200 may sense the orientation of the user's headand/or gaze direction of the user at a predetermined time interval(e.g., 2, 4, 6, 10 or more times per second) and provide an output tothe processing device.

The processing device may determine the time duration that user 2201looks in a direction of the graphical icon based on whether the sensedviewing direction of the user's head and/or gaze direction changes fromthe previously detected viewing direction. For example, if the currentlydetected viewing direction stays unchanged from the previously detectedviewing direction (e.g., at the last time interval), the processingdevice may increase the amount of time that the user has looked in thedirection of the graphical icon by the predetermined time interval. Ifthe currently detected viewing direction changes from the previouslydetected orientation, the processing device may determine that the userhas moved the direction of his or her sight line away from the graphicalicon and may reset the amount of elapsed time that the user has lookedin the graphical icon to zero. If the time duration that the user 2201looks in a direction of the graphical icon is less than a predeterminedtime threshold, the processing device may determine that user 2201 doesnot intend to initiate control of the on-board component and may nottake any action. On the other hand, if the time duration that theviewing direction of user 2201 stays in a detected viewing direction isgreater than or equal to the predetermined time threshold, theprocessing device may determine that user 2201 intends to controloperation of the on-board component and may initiate the control of theon-board component.

In some embodiments, as discussed above, alternative or in addition tomonitoring an orientation of the user's head, the processing device ofAR device 200 may be configured to monitor a user's gaze direction basedon output received from one or more eye tracking sensors. For example,the processing device may be configured to determine whether user 2201is looking in a direction of the graphical icon based on the detectedgaze direction of user 2201.

FIG. 23A illustrates an example of a graphical icon associated withcontrolling an on-board component of an AR device consistent withdisclosed embodiments. For example, rendering services module 370,discussed above in reference to FIG. 3, may store software instructionsthat, when executed by a processing device (e.g., microprocessor 208),cause a graphical icon associated with controlling one or more aspectsof an on-board component to be displayed.

As shown in FIG. 23A, a graphical icon 2301 associated with control ofan on-board component is displayed on display 204 of AR device 200. Insome embodiments, the on-board component of AR device 200 may include anon-board camera (e.g., camera 203), and the graphical icon displayed ondisplay 204 may be associated with operations of the on-board camera.For example, graphical icon 2301 may be associated with a power-onfunction of the on-board camera, an image capture function of theon-board camera, and/or other camera functionality. It should beunderstood that while a graphical icon is used in connection withcontrol of the on-board component, text, symbols, graphics, and/or acombination thereof may be used for controlling operation of theon-board component without departing from the spirit of the presentdisclosure.

Graphical icon 2301 may be shown at a predetermined position on display204, such as the center of display 204, an upper portion of display 204,or any other suitable location. The predetermined location fordisplaying graphical icon 2301 may be pre-configured by user 2201through a user interface of AR device 200. For example, display 204 mayallow user 2201 to select a location of display 204 at which graphicalicon 2301 will be displayed. In some embodiments, after graphical icon2301 is displayed at a pre-set location, user 2201 may be allowed todrag graphical icon 2301 to another location on display 204 that ispreferred by user 2201. For example, the processing device of AR device200 may set the new location as a default location for displayinggraphical icon 2301, and display graphical icon 2301 at the new locationwhen AR device 200 receives user input to display the graphical iconassociated with control of the on-board component. In other embodiments,the location of graphical icon 2301 may be locked to a predeterminedposition of display 204 such that user 2201 cannot reposition thegraphical icon.

After graphical icon 2301 associated with control of the on-boardcomponent is provided on display 204, AR device 200 may be configured todisplay graphical icon 2301 in a fixed location relative to real worldcoordinates. For example, the user may perceive that graphical icon 2301is located at a particular location in the real world, and when the usermoves or looks away from that particular location, graphical icon 2301may not appear on the display. That is, the display of graphical icon2301 may depend not only on the specific viewing direction of the user,but also the physical location of the user. Even if the viewingdirection of the user remains the same, if the physical location of theuser changes, graphical icon 2301 may disappear from the display.Graphical icon 2301 may reappear on the display when the user looksagain at the particular location in the real world.

In other embodiments, graphical icon 2301 may be provided at a fixed,perceived location relative to the user, such as in an unlocked mode.For example, graphical icon 2301 may appear on the display whenever thehead and/or eyes of user 2201 is moved to a particular viewing direction(e.g., a certain degree to the right direction, regardless the physicallocation of the user in the real world). Once graphical icon 2301 isdisplayed, the location of graphical icon 2301 may be changed as theuser rotates his or her head and/or eyes toward the icon. For example,once graphical icon 2301 is shown on the display, subsequent head and/oreye movements further in the upward direction may cause the location ofgraphical icon 2301 to move downward on the display. Similarly,subsequent head and/or eye movements in a downward direction may causethe location of graphical icon 2301 to move upward on the display.Additionally, the processing device may be configured to also causechanges in the left-right position of the graphical icon relative to thedisplay in response to detected horizontal movements of the user's headand/or eyes (e.g., right-left changes in the looking direction of theuser's head and/or eyes).

FIG. 23B illustrates an example of accessing a graphical icon displayedby an AR device that is associated with controlling an on-boardcomponent consistent with disclosed embodiments. As shown in FIG. 23B, areticle 2302 may be shown on display 204. While a cross-hair shape isused in FIG. 23B, reticle 2302 may have any suitable shape. For example,reticle 2302 may be represented as a cross shape, a dot, a circle, asquare, an arrow, etc. In some embodiments, the reticle may berepresented on the display as one or more icons, still or video images,or text. Various aspects of the reticle may be changed by user input.For example, the size, color, shape, orientation, or any other attributeassociated with the reticle may be changed according to user inputprovided via a user interface of AR device 200.

Reticle 2302 may constitute a virtual reticle shown on display 204 whoseposition on display 204 may be changed in response to user input. Forexample, one or more eye tracking sensors, as described above, mayenable tracking of a user's gaze direction, and the position of reticle2304 on display 204 may be changed with determined changes in the user'sgaze direction.

In some embodiments, reticle 2302 may be fixed relative to display 204(e.g., may be physical inscribed or otherwise marked on display 204 orany other suitable component of AR device 200) such that reticle 2302moves with changes in the user's head orientation and/or direction ofthe user's gaze. In such embodiments, reticle 1001 may be used, e.g., toselectively overlap with and select portions of graphical icon 2301 thatremains unlocked relative to display 204 and fixed relative to realworld coordinates.

For example, if the processing device determines that the position ofreticle 2302 is moved to intercept a portion of graphical icon 2301, theprocessing device may cause graphical icon 2301 to be highlighted ondisplay 204. By highlighting graphical icon 2301, the processing deviceof AR device 200 may provide an indication to user 2201 that graphicalicon 2301 may be selected if the user continues looking in the directionof graphical icon 2301 for a certain period of time. In someembodiments, if the processing device determines that the position ofreticle 2302 is moved to intercept a portion of graphical icon 2301, theprocessing device may cause aspects of the reticle to be changed toprovide an indication to user 2201 that graphical icon 2301 may beselected by further user actions, such as looking in the direction ofgraphical icon 2301 for a certain period of time. For example, the colorof reticle 2302 may be changed when the position of reticle 2302 ismoved to intercept a portion of the graphical icon 2301.

In some embodiments, reticle 2302 may be shown at the center of display204 when it is initially turned on, and may be moved to other locationsin response to user input. For example, user 401 may drag reticle 2302from one position to another position on display 204 by touching display204. Alternatively or additionally, the processing device may beconfigured to move the position of reticle 2302 in response to sensedchanges in the user's gaze direction (e.g. using eye tracking, asdiscussed above). Further, as discussed above, the position of reticle2302 relative to graphical icon 2301 may be changed by tracking theuser's viewing direction (e.g., head orientation and/or gaze direction0and updating the position on display 204 at which graphical icon 2301 isdisplayed as the user's viewing direction changes. The processing devicemay move the position of graphical icon 2301 the opposite direction asthe detected direction of the movement of the head and/or eyes of user401, such that reticle 2302 intercepts graphical icon 2301.

Once reticle 2302 is moved into a position that intercepts a portion ofgraphical icon 2301, the processing device of AR device 200 may causegraphical icon 2301 to be highlighted on display 204. Subsequently, user2201 may cause graphical icon 2301 to be selected by looking in thedirection of graphical icon 2301. Alternatively, user 2201 may causegraphical icon 2301 to be selected by pressing the portion of thedisplay at which reticle 2302 is located for a certain period of time.In some embodiments, user 2201 may cause graphical icon 2301 to beselected by pressing a physical button that is placed on AR device 200.

If the processing device of AR device 200 detects that graphical icon2301 is selected (e.g., by determining that user 2201 is looking in adirection of the graphical icon), the processing device may initiatecontrol of certain aspect of an on-board component. For example, theon-board component may include an on-board camera, and the processingdevice may cause the on-board camera to be powered on in response to thedetected selection of graphical icon 2301. As another example, theprocessing device may cause the image capture function of the on-boardcamera to be turned on in response to the detected selection ofgraphical icon 2301. In another example, the processing device may causean image to be captured by the on-board camera on in response to thedetected selection of graphical icon 2301.

In some embodiments, the on-board component may include an on-boardmicrophone, and a graphical icon associated with operations of themicrophone may be displayed on display 204. For example, the processingdevice of AR device 200 may cause the on-board microphone to be poweredon in response to the detected selection of graphical icon. As anotherexample, the processing device may cause the audio recording function ofthe on-board microphone to be turned on in response to the detectedselection of the graphical icon.

In some embodiments, in response to a detected selection of a graphicalicon that is associated with control of an on-board component, theprocessing device of AR device 200 may cause a menu of control icons tobe shown on the display. The menu of control icons may includeoperations associated with different aspects of the on-board component,thereby allowing the user to control a particular aspect of the on-boardcomponent as desired.

FIG. 23C illustrates an example of a menu of operations associated withan on-board camera that is displayed by an AR device consistent withdisclosed embodiments. For example, rendering services module 370,discussed above in reference to FIG. 3, may store software instructionsthat, when executed by a processing device (e.g., microprocessor 208),cause a control menu to be displayed. The control menu may include text,symbols, graphics, and/or a combination thereof.

As shown in FIG. 23C, in response to the detected selection of graphicalicon 2301 (e.g., by determining that user 2201 is looking in a directionof the graphical icon), the processing device of AR device 200 may causea camera control menu 2303 to be displayed on display 204 of AR device200. The camera control menu 2303 may include text, icons, and/orsymbols associated with one or more of shutter speed, aperture,International Standards Organization (ISO), flash, and/or otherfunctions of the on-board camera. Camera control menu 2303 may besemi-transparent such that it is layered on top of the user's field ofview and does not completely block the user's field of view. In someembodiments, camera control menu 2303 may be displayed at a location onthe display that is the same or near to the location where graphicalicon 2301 was displayed such that user 2201 does not need to move his orher sight line drastically in order to make further selections on cameracontrol menu 2303.

After camera control menu 2303 is displayed, the processing device of ARdevice 200 may cause a particular aspect of the on-board camera to becontrolled if the processing device determines that user 2201 is lookingin a direction intersecting a particular item shown on camera controlmenu 2303. For example, if the processing device determines that user2201 is looking in a direction intersecting menu item “flash” on cameracontrol menu 2303, the processing device may cause the flash function ofthe on-board camera to be turned on. As another example, if theprocessing device determines that user 2201 is looking in a directionintersecting menu item “flash” on camera control menu 2303 and the flashfunction of the camera is currently turned on, the processing device maycause the flash function of the on-board camera to be turned off. Insome embodiments, user 2201 may cause a particular item shown on cameracontrol menu 2303 to be selected by moving a reticle to the location ofthe particular item, touching the portion of the display where the itemis shown, pressing a physical button of the device, or the like.

FIG. 24 is a flowchart of an exemplary process 2400 for controllingoperation of an on-board component by an AR device consistent withdisclosed embodiments. As an example, one or more steps of process 2400may be performed by AR device 200 shown in FIG. 2. For example, inprocess 2400, a sensor of AR device 200 may detect a viewing direction(e.g., head orientation or gaze direction) of user 401 and provide acorresponding output to a processing device of AR device 200.

At step 2410, AR device 200 may cause a graphical icon associated withcontrolling an aspect of an on-board component to be shown on display204. For example, the processing device of AR device 200 may executeinstructions of rendering services module 370 to perform this step. Theon-board component of AR device 200 may include a camera. The graphicalicon may be associated with a power-on function of the on-board camera,an image capture function of the on-board camera, or any other functionsof the on-board camera.

The graphical icon may be displayed in a manner such that the userperceives the location of the graphical icon as fixed relative to realworld coordinates. In some embodiments, the processing device may beconfigured to update a position at which the graphical icon is shown onthe display based on the output of the sensor. For example, the locationof the graphical icon shown on the display may be changed by a detectedmovement in the orientation of the head of the user and/or by a detectedmovement in the gaze of the user's eyes.

At step 2420, AR device 200 may determine, based on the output of thesensor, whether user 2201 is looking in a direction of a selectableelement of the graphical icon. For example, the sensor may be configuredto provide an output indicative of the viewing direction of the user bytracking a pattern of movement of an orientation of the head of theuser. As another example, the sensor may be configured to provide anoutput indicative of the viewing direction of the user by tracking agaze of the user's eyes. For example, the processing device of AR device200 may execute instructions of positional processing module 378 toperform this step. The processing device may determine the direction ofthe user's sight line based on the viewing direction of user 2201. Theprocessing device may further use coordinates of multiple referencespoints of AR device 200 received from the sensors to determine thedirection of the user's sight line.

At step 2430, AR device 200 may control an aspect of the on-boardcomponent when user 2201 is determined to be looking in the direction ofthe graphical icon. Further, in some embodiments, AR device 200 mayinitiate control of an aspect of the on-board component after the userhas looked in the direction of the graphical icon for a predeterminedperiod of time (e.g., 1 second, 2 second, 3 seconds, 5 seconds, 10seconds, ect.) The processing device of AR device 200 may executeinstructions of rendering services module 370, command processor 366,and/or third party applications 362 to perform this step. The on-boardcomponent may include an on-board camera, and the control of the aspectof the on-board component may include powering on the on-board camera,capturing an image using the on-board camera, and so on. In someembodiments, a menu of camera control icons may be shown on the displaywhen user 2201 is determined to be looking in the direction of graphicalicon. The menu of camera control icons may include icons associated withone or more of shutter speed, aperture, ISO, flash, and so on.

Sharing Information Between Users of AR Devices

In some embodiments, a one-to-many system may enable sharing ofinformation between multiple wearable devices. For example, informationavailable to a particular wearable device user may be made available tomultiple other users by showing that information on the displays of themultiple users.

For example, consistent with an embodiment, a system may interact withand share information among a plurality of users of a correspondingplurality of wearable devices. The system may comprise a networkinterface and at least one processing device associated with the system.The at least one processing device may be configured to receive, via thenetwork interface, information from a first wearable device; select fromthe plurality of wearable devices one or more designated wearabledevices to receive the information; and send, via the network interface,the information to the one or more designated wearable devices.

FIG. 25 illustrates an example environment 2500 for sharing informationbetween users of AR devices consistent with disclosed embodiments. Asshown in FIG. 25, users 2501-2505 each wears an AR device 2506-2510,respectively. AR devices 2506-2510 may include similar components asthose of AR device 200 shown in FIG. 2, while each of the AR devices2506-2510 may be implemented with different hardware components and/orsoftware configurations. AR devices 2506-2510 may communicate withserver system 110 via network 130 to share information with one another.Users may share any kind of information (e.g., text, images, audio,and/or video) with one another. For example, user 2501 may share animage retrieved by AR device 2506 with users 2502 and 2503 by sendingthe image to AR devices 2507 and 2508 via server system 110 and network130. As another example, user 2501 may share an audio recordingretrieved by AR device 2506 with users 2504 and 2505 by sending theaudio recording to AR devices 2509 and 2510 via server system 110 andnetwork 130.

Server system 110 may include a network interface for communicating withthe AR devices via network 130. For example, server system 110 mayinclude an Internet connection, a cellular connection, a WAN connection,a LAN connection, and/or any other types of network connections that canbe used for communication. Server system 110 may receive informationfrom one or more AR devices and send information to one or more ARdevices via the network interface.

In some embodiments, when the same information is to be sent to multipleAR devices, server system 110 may be configured to send the informationto the AR devices simultaneously through broadcasting or multicastingmechanisms. For example, server system 110 may broadcast certaininformation on a given time and/or frequency resource such that ARdevices 2505-2510 may receive the information at a substantially sametime. As another example, server system 110 may multicast certaininformation to a particular set of AR devices on a given time and/orfrequency resource such that the particular set of AR devices mayreceive the information at a substantially same time, while other ARdevices may not receive the information. In other embodiments, serversystem 110 may be configured to send the information to each of ARdevices 2505-2510 individually at a different time and/or usingdifferent frequency resources.

Server system 110 may include one or more processing devices configuredto execute software instructions programmed to facilitate informationsharing between users 2501-2505. For example, the processing devices mayexecute software instructions to identify a group of users that sharesone or more common user attributes, such as a geographic location, anaffiliation with certain organization, a hobby, etc. The processingdevices may include one or more suitable hardware-based processingdevices, such as a microprocessor, controller, central processing unit,etc. that can execute software-based instructions. Such softwareinstructions may be stored, for example, in memory devices (e.g., memory112). In some embodiments, the software instructions for performinginformation sharing between users of AR devices may be stored in acomputer-readable storage medium, and/or in a database co-located withor remotely located from server system 110.

In some embodiments, server system 110 may include one or more databases(e.g., database 113) for storing information of users of AR devices. Forexample, server system 110 may include databases for storing profiles ofusers of AR devices. The user profiles may include information of userssuch as name, age, location, profession, interests, contacts, and so on.The user profiles may also include user preferences related toinformation sharing. For example, user 2501 of AR device 2506 may setcertain information to be public and accessible to other users. On theother hand, user 2501 may set other information to be private and onlyaccessible to a group of designated users, such as users in a contactlist of user 2501. In some embodiments, the user profiles may includeuser activity data, for example, a history of user activity, such asinteractions with other users of AR devices, information retrieved bythe user from the server system, locations that the user went to, daysand/or times during which the AR device was used by the user, or thelike, collected over a certain period of time (e.g., the past week,month, three months, or any other suitable time period).

As shown in FIG. 25, AR device 2506 may send certain information toserver system 110 via network 130 in response to a request received fromuser 2501 to share the information. For example, user 2501 may initiateAR device 2506 to capture an image of the user's current field of viewand instruct AR device 2506 to share the image with other users of ARdevices. In some embodiments, a graphical icon or text command may beshown on the display of AR device 2506 after the image is captured,allowing user 2501 to share the image with other users. In otherembodiments, a virtual menu may be shown on the display of AR device2506 after the image is captured, and user 2501 may initiate sharing ofthe image by selecting a corresponding item on the virtual menu. Afterdetecting the selection of the graphical icon, text command, or menuitem, AR device 2506 may send the captured image to server system 110for sharing with other users.

In some embodiments, user 2501 may request that AR device 2506 sharecertain information before the particular information is acquired by theAR device. For example, user 2501 may request that AR device 2506 sharean image of the user's current field of view before the image iscaptured. User 2501 may select a graphical icon or text command on thedisplay of AR device 2506, indicating a request to share an image of theuser's current field of view with other users. In response, AR device2506 may cause the image to be captured and then send the captured imageto server system 110 for sharing with other users without requiringfurther input from user 2501.

In some embodiments, user 2501 may designate one or more users forsharing the information with, and AR device 2506 may send informationfrom the designated users to server system 110 along with theinformation to be shared. For example, user 2501 may select one or moreusers from the contact list to receive the shared information, and ARdevice 2506 may send identifiers of the selected users to server system110 along with the information to be shared. As another example, ARdevice 2506 may retrieve identifiers of the AR devices corresponding tothe selected users, and send identifiers of the AR devices to serversystem 110 along with the information to be shared. As another example,user 2501 may input names of the users to receive the sharedinformation. In response, AR device 2506 may retrieve identifiers of theusers corresponding to the names, and send the user identifiers toserver system 110 along with the information to be shared.

After receiving the shared information from AR device 2506, serversystem 110 may select one or more designated AR devices to receive theshared information. For example, if user identifiers are provided by ARdevice 2506 along with the shared information, server system 110 mayretrieve, from the database, information of the corresponding AR devicesbased on the received user identifiers, and send the shared informationto the corresponding AR devices. As another example, if identifiers ofthe AR devices for receiving the shared information are provided by ARdevice 2506, server system 110 may send the shared information to thecorresponding AR devices without having to look up the information ofthe AR devices in the database. In another example, if the sharedinformation is marked as public and may be shared with all users of ARdevices or all users of AR devices belonging to, for example, aparticular user group, server system 110 may retrieve, from thedatabase, information of the users and the corresponding AR devices thathave subscribed to receive information from user 2501, and send theshared information to the corresponding AR devices.

In some embodiments, if user 2501 does not designate any users toreceive the shared information, a default setting may be applied forselecting the users to receive the shared information. For example, ifuser 2501 does not designate any users to receive the sharedinformation, server system 110 may determine that the information may bepublicly shared, and thus, any users of AR devices may receive theinformation. As another example, if user 2501 does not designate anyusers to receive the shared information, server system 110 may determinethat the information may be shared among users who are listed ascontacts of user 2501, and thus, only users that are contacts of user2501 may receive the information. Alternatively, if user 2501 does notdesignate any users to receive the shared information, server system 110may share the information among the users that were previouslydesignated by user 2501 as users to share information with in a previousrequest by user 2501. In some embodiments, if user 2501 does notdesignate any users to receive the shared information, server system 110may send a message to AR device 2506 requiring information of thedesignated users to receive the shared information. AR device 2506 maythen display a message on the display, prompting user 2501 to provideinput as to the users to receive the shared information.

In some embodiments, if user 2501 does not designate any users toreceive the shared information, server system 110 may analyze the userprofile of user 2501 and other users of AR devices, and select the usersto receive the shared information based on the user profiles. Forexample, server system 110 may identify a commonality between user 2501and other users, and send the shared information to the users that havethe commonality. The commonality may be a location, an affiliation, aprofession, an interest, and/or any other attributes that can beidentified based on profiles of the users. In some embodiments, inaddition to sending the shared information to the users that have acommonality with user 2501, server system 110 may also send a message touser 2501 including information of the users that receive the sharedinformation, such as name and/or location of the users, and theidentified commonality between user 2501 and the users that receive theshared information.

In some embodiments, a combination of commonalities may be used toselect users to receive the shared information. For example, serversystem 110 may identify users that share both the same location and sameinterest as that of user 2501 and send the received information to theseusers. Referring to FIG. 25, for example, after receiving the sharedinformation from user 2501, server system 110 may analyze the userprofiles of users 2501-2505, identify that user 2502 and user 2503 sharethe same location and interest with user 2501, and then send thereceived information to user 2502 and user 2503.

After receiving the shared information, each of the AR devices may causethe information to be shown on a display such that the sharedinformation can be accessible by the users of the AR devices. Forexample, the shared information may be displayed at a predeterminedlocation on the display of the AR devices. The predetermined location atwhich the shared information is displayed may be pre-configured by theusers of the AR devices. In some embodiments, the shared information maybe shown on a portion of the display of the AR device so that the sharedinformation does not block the view of the user. In other embodiments,AR device may display the shared information on the display such thatthe shared information is semi-transparent and the user can view his orher environment behind the shared information. When the sharedinformation includes an audio recording, each of the AR devices maycause the audio recording to be played by the AR devices. In otherembodiments, real time video from one user may be streamed to a portionof the display of the AR devices of one or more other users.

In some embodiments, a notification may be first displayed on thedisplay of the AR device prior to the display of the shared information,notifying the user receipt of the shared information. The notificationmay also indicate an identity of the sending user. In some embodiments,the notification may indicate certain features of the sharedinformation, such as the type of the shared information, the size of theshared information, the time at which the shared information wasreceived, or the like. In some embodiments, along with the notification,the display of the AR device may also show operations associated withthe shared information that the user may select to perform. For example,an option may be shown on the display allowing the user to access theshared information. As another example, an option may be shown on thedisplay allowing the user to access the shared information at a latertime. In another example, an option may be shown on the display allowingthe user to delete the shared information without viewing theinformation. It should be understood that the AR devices that receivethe shared information may have different hardware and/or softwareconfigurations, and the shared information and/or the notification maybe displayed in a same or different manner in each of the AR devices.

In some embodiments, another user may respond to the shared informationwith audio and/or images related to the shared information. For example,a user of an AR device may send an image (e.g., an image of an engine)to another person using anther device (e.g., a similar or different ARdevice, or another computing device, such as a smartphone, tablet, etc.)who is located elsewhere and who may respond with screen annotationsand/or audible instructions to help guide the user. The screenannotations and/or audible instructions may assist the user (e.g.,providing instructions to the user related to the engine).

In some embodiments, a user of the AR device may also request to accessinformation associated with another user via server system 110 andnetwork 130. For example, user 2501 may wish to access an image of thecurrent field of view of user 2502, and user 2501 may request user 2502to share the image via server system 110 and network 130. In response toa user input from user 2501, AR device 2506 may send a request to serversystem 110, requesting user 2502 to share an image of the user's currentfield of view. Server system 110 may forward the request to AR device2507 and inquire whether user 2502 will accept the request to share theimage. After receiving the request, AR device 2507 may cause a messageto be shown on the display, indicating to user 2502 that a request toshare the image is received from user 2501 and inquiring whether user2502 will accept the request. If user 2502 provides an input rejectingthe request, AR device 2507 may send a response to server system 110,indicating that the request was rejected, and server system 110 maysubsequently forward the response to AR device 2506. If user 2502provides an input accepting the request, AR device 2507 may capture animage of the current field of view of user 2502 and send the image toserver system 110. Server system 110 may subsequently forward the imageto AR device 2506, which may in turn cause the image to be shown on thedisplay of AR device 2506.

In some embodiments, a user of the AR device may set a degree of sharingwith respect to certain information stored on the AR device. Forexample, user 2501 may select a picture folder on the AR device to beshared with a designated group of users, e.g., user 2502 and user 2503.As a result, when a new picture is added to the picture folder, ARdevice 2506 may automatically send the picture to server system 110 andrequest the picture to be shared with user 2502 and user 2503. Serversystem 110 may then send the picture to AR devices 2507 and 2508according to the received request from AR device 2506. Similarly, whenan existing picture in the picture folder is changed or updated, ARdevice 2506 may automatically send the updated picture to server system110 and request the updated picture to be shared with user 2502 and user2503. Server system 110 may then send the updated picture to AR devices2507 and 2508 according to the received request from AR device 2506.

As another example, user 2501 may select a picture folder on the ARdevice to be shared with users that have the same location and interestwith user 2501. When a new picture is added to the picture folder, ARdevice 2506 may automatically send the picture to server system 110 andrequest the picture to be shared with users that have the same locationand interest with user 2501. Server system 110 may then identify theusers of AR devices that have the same location and share one or moreinterests with user 2501 based on the user profiles, and send thepicture to the corresponding AR devices. Similarly, when an existingpicture in the picture folder is changed or updated, AR device 2506 mayautomatically send the updated picture to server system 110 and requestthe updated picture to be shared with users that have the same locationand one or more shared interests with user 2501. Server system 110 maythen identify the users of AR devices that have the same location andshared interests with user 2501, and send the updated picture to thecorresponding AR devices. It should be understood that while in thisexample location and interest are used as target user attributes inselecting users to receive the shared information, any other attributesassociated with the users may be used for the selection of users.

In some embodiments, shared information from multiple users may beorganized on the display of an AR device. For example, sharedinformation may be shown on the display of an AR device in a hierarchy(e.g., a scrollable list of selectable images, videos, and/or audioicons). The hierarchy of available information may be shown, forexample, on a side of the display or along a top of the display of theAR device.

FIG. 26 is a flowchart of an exemplary process 2600 for sharinginformation between users of AR devices consistent with disclosedembodiments. As an example, one or more steps of process 2600 may beperformed by server system 110 shown in FIG. 1.

At step 2610, server system 110 may receive, via a network interface,information from a first wearable device, such as AR device 200. Thenetwork interface may include one or more of an Internet connection, acellular connection, a WAN connection, a LAN connection, and/or anyother types of network connections. The information may include imagedata retrieved from an image capture device associated with the firstwearable device, audio data retrieved from an audio device associatedwith the first wearable device, text messages retrieved from an inputdevice associated with the first wearable device, and/or a combinationthereof.

At step 2620, server system 110 may select from a plurality of wearabledevices one or more designated wearable devices to receive theinformation. For example, the processing device of server system 110 maybe configured to select the one or more designated wearable devicesbased on a predetermined setting retrieved over the network interface.As another example, the processing device may select the one or moredesignated wearable devices based on an analysis of user profiles of thecorresponding plurality of users of the plurality of wearable devices.In some embodiments, the analysis of user profiles may includeidentifying a commonality of the plurality of users based on the userprofiles.

At step 2630, server system 110 may send, via the network interface, theinformation to the one or more designated wearable devices. In someembodiments, the processing device of server system 110 may furthercause the information sent to the designated devices to be shown on adisplay associated with each of the designated devices.

FIG. 27 is a flowchart of an exemplary process 2700 for selectingdesignated AR devices to receive information consistent with disclosedembodiments. As an example, one or more steps of process 2700 may beperformed by server system 110 shown in FIG. 1.

At step 2710, server system 110 may receive user profiles of a pluralityof users of corresponding AR devices. For example, the processing deviceof server system 110 may receive the user profiles from each of thecorresponding AR devices via the network interface. As another example,the processing device of server system 110 may receive the user profilesfrom a database that is co-located with or remotely located from serversystem 110. In another example, the processing device of server system110 may receive the user profiles from a third-party system that storesinformation of the user profiles.

At step 2720, server system 110 may identify a commonality of theplurality of users based on user profiles. For example, the processingdevice of server system 110 may identify a common user attribute, suchas a location, an affiliation, a profession, an interest, and/or anyother attributes that can be identified based on the user profiles,among the plurality of users. In some embodiments, the processing devicemay identify more than one commonality among the plurality of users. Forexample, the processing device may identify both a location and aninterest as commonalities shared by a subset of the plurality of usersbased on the user profiles.

At step 2730, server system 110 may select one or more designateddevices to send information to based on the commonality. For example,the processing device of server system 110 may select the devicescorresponding to users that share a certain commonality with aparticular user (e.g., the user of the sending device that provides theinformation) as the receiving devices to send information to. As anotherexample, the processing device may select the devices corresponding tousers that share a combination of commonalities with a particular user(e.g., the user of the sending device that provided the information). Insome embodiments, the required commonality may be indicated by thesending device that provides the information, and the processing deviceof server system 110 may select the devices corresponding to users whoshare the required commonality.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks, floppydisks, or CD ROM, or other forms of RAM or ROM, USB media, DVD, Blu-rayor other optical drive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets. One or more of such software sections or modules can beintegrated into a computer system or existing e-mail or browsersoftware.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

1. A wearable device for controlling operation of an on-board component,the wearable device comprising: a display; at least one sensorconfigured to provide an output indicative of a viewing direction of auser; and at least one processing device configured to: cause at leastone graphical icon associated with a control of at least one aspect ofthe on-board component to be shown on the display such that the userperceives the at least one graphical icon to be located at a location inthe real world and fixed relative to coordinates of the real worldlocation; cause a reticle to be displayed on the display; move, based onthe output of the at least one sensor, the reticle to intercept aportion of the at least one graphical icon; change, based on the reticlehaving intercepted the portion of the at least one graphical icon, atleast one aspect of the reticle to provide a visual indication to theuser that the at least one graphical icon can be selected by looking inthe direction of the at least one graphical icon for a predeterminedperiod of time; select the at least one graphical icon when it isdetermined that the user has been looking in the direction of the atleast one graphical icon for the predetermined period of time; and causethe on-board component to capture an image of a field of view of theuser when the at least one graphical icon is selected.
 2. The wearabledevice of claim 1, wherein the on-board component includes a camera. 3.The wearable device of claim 2, wherein the at least one graphical iconis associated with a power-on function of the on-board camera.
 4. Thewearable device of claim 2, wherein the at least one graphical icon isassociated with an image capture function of the on-board camera.
 5. Thewearable device of claim 2, wherein the at least one processing deviceis configured to show a menu of camera control icons on the display. 6.The wearable device of claim 5, wherein the menu of camera control iconscomprises icons associated with one or more of shutter speed, aperture,ISO, and flash.
 7. The wearable device of claim 1, wherein the at leastone processing device is configured to update a position at which the atleast one graphical icon is shown on the display based on the output ofthe at least one sensor.
 8. The wearable device of claim 1, wherein theat least one sensor is configured to provide an output indicative of theviewing direction of the user by tracking a pattern of movement of anorientation of a head of the user.
 9. The wearable device of claim 1,wherein the at least one sensor is configured to provide an outputindicative of the viewing direction of the user by tracking a gaze ofthe user's eyes.
 10. A method for controlling operation of an on-boardcomponent of a wearable device, the method comprising: causing at leastone graphical icon associated with a control of at least one aspect ofthe on-board component to be shown on a display of the wearable devicesuch that the user perceives the at least one graphical icon to belocated at a location in the real world and fixed relative tocoordinates of the real world location; causing a reticle to bedisplayed on the display; moving, based on an output of at least onesensor of the wearable device configured to provide an output indicativeof a viewing direction of the user of the wearable device, the reticleto intercept a portion of the at least one graphical icon; changing,based on the reticle having intercepted the portion of the at least onegraphical icon, at least one aspect of the reticle to provide a visualindication to the user that the at least one graphical icon can beselected by looking in the direction of the at least one graphical iconfor a predetermined period of time; selecting the at least one graphicalicon when it is determined that the user has been looking in thedirection of the at least one graphical icon for the predeterminedperiod of time; and causing the on-board component to capture an imageof a field of view of the user when at least one graphical icon isselected.
 11. The method of claim 10, wherein the on-board componentincludes a camera.
 12. The method of claim 11, wherein the at least onegraphical icon is associated with a power on function of the on-boardcamera.
 13. The method of claim 11, wherein the at least one graphicalicon is associated with an image capture function of the on-boardcamera.
 14. The method of claim 11, wherein a menu of camera controlicons is shown on the display.
 15. The method of claim 14, wherein themenu of camera control icons comprises icons associated with one or moreof shutter speed, aperture, ISO, and flash.
 16. The method of claim 10,further comprising updating a position at which the at least onegraphical icon is shown on the display based on the output of the atleast one sensor.
 17. The method of claim 10, further comprisingmonitoring a pattern of movement of an orientation of a head of theuser.
 18. The method of claim 10, further comprising monitoring a gazeof the user's eyes.
 19. A non-transitory computer-readable storagemedium storing program instructions executable by at least one processorto perform a method for controlling operation of an on-board componentof a wearable device, the method comprising: causing at least onegraphical icon associated with a control of at least one aspect of theon-board component to be shown on a display of the wearable device suchthat the user perceives the at least one graphical icon to be located ata location in the real world and fixed relative to coordinates of thereal world location; causing a reticle to be displayed on the display;moving, based on an output of at least one sensor of the wearable deviceconfigured to provide an output indicative of head orientation of theuser of the wearable device, the reticle to intercept a portion of theat least one graphical icon; changing, based on the reticle havingintercepted the portion of the at least one graphical icon, at least oneaspect of the reticle to provide a visual indication to the user thatthe at least one graphical icon can be selected by looking in thedirection of the at least one graphical icon for a predetermined periodof time; selecting the at least one graphical icon when it is determinedthat the user has been looking in the direction of the at least onegraphical icon for the predetermined period of time; and causing theon-board component to capture an image of a field of view of the userwhen the at least one graphical icon is selected.